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1.0  Introduction 


This  special  report  outlines  the  steps  in  planning  and  implementing  advanced  distributed 
simulation  (ADS)-based  testing.  The  methodology  is  divided  into  two  parts:  an  ADS-inclusive 
test  concept  development  methodology  and  an  ADS-based  test  planning  and  implementation 
methodology. 

The  objective  of  the  test  concept  development  methodology  is  to  determine  whether  ADS-based 
testing  is  either  required  (because  conventional  testing  cannot  fully  represent  the  operational 
environment)  or  desired  (because  there  are  benefits  from  ADS-based  testing,  such  as  cost  or  time 
savings).  If  the  decision  is  made  that  a  conventional  (i.e.,  nondistributed)  testing  approach  will 
suffice,  traditional  testing  methodologies  are  used,  and  the  section  on  the  ADS-based  test 
planning  and  implementation  methodology  does  not  apply. 

If  the  decision  is  made  to  implement  ADS-based  testing,  the  ADS-based  test  planning  and 
implementation  methodology  should  be  followed.  This  methodology  tailors  the  steps  given  in 
the  Defense  Modeling  and  Simulation  Office  (DMSO)  High  Level  Architecture  (HLA) 
Federation  Development  and  Execution  Process  (FEDEP)  model  [Ref.  1]  to  test  and  evaluation 
(T&E)  applications. 

The  processes  are  intended  to  serve  as  checklists  and  “how  to”  guides.  The  intent  is  to  provide  a 
logical  framework  for  assessing  potential  uses  of  ADS  within  a  given  program  and  to  offer 
guidelines  for  ADS  planning  activities  once  a  decision  to  use  ADS  is  made. 

Linking  comes  with  certain  costs.  There  is  usually  some  degradation  of  data  when  they  are 
shipped  around,  and  there  are  the  dollar  and  time  costs  associated  with  creating  a  linked 
environment.  Because  of  the  costs,  from  a  tester’s  perspective,  there  have  to  be  reasons  to  link. 
The  most  compelling  reason  for  using  ADS  in  support  of  testing  is  the  complexity  of  the  test 
environment.  Complex  environments  with  many  players  and  many  interactions  are  too 
expensive  to  represent  in  the  open  air  test  arena.  (The  complex  environments  may  also  be  too 
expensive  to  represent  in  newly  built  stand-alone  models.)  When  a  test  environment 
encompasses  many  interactions  with  human  decision  making  involved,  models  tend  to  have 
serious  credibility  problems.  If  a  system  under  test  requires  a  highly  complex,  dynamic,  human- 
in-the-loop  environment,  there  is  a  good  chance  that  ADS  is  a  technology  which  can  benefit  the 
test  program. 

This  report  offers  a  methodology  to  incorporate  ADS  as  a  factor  in  test  concept  development  and 
detailed  planning.  The  costs  and  the  benefits  associated  with  the  use  of  ADS  are  always  very 
program  specific.  Test  planners  will  have  to  look  at  the  details  of  their  particular  program  to  see 
potential  applications,  costs,  and  benefits.  One  cautionary  note  about  costing  should  be  borne  in 
mind.  The  benefits  of  better  testing  may  be  reflected  in  better  products  and  more  efficient 
production.  Cost  benefit  analyses  should  take  a  program-level  perspective  and  not  be  limited 
exclusively  to  test  costs. 
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2.0  ADS-Inclusive  Test  Concept  Development  Methodology 


The  methodology  described  in  this  paper  uses  an  example  which  is  couched  in  terms  of 
operational  test  and  evaluation  (OT&E).  But,  as  OT&E  moves  left  on  the  acquisition  timeline 
and  as  new  systems  demand  ever  more  complex  test  environments,  the  process  is  applicable  to 
developmental  test  and  evaluation  (DT&E)  as  well. 

Historically,  T&E  planners  have  had  to  live  with  severe  test  asset  and  cost  limitations.  As  a 
result,  the  tendency  during  test  concept  development  has  been  to  analyze  the  operating 
environment  from  the  bottom  up.  Typically,  the  process  involved  identifying  the  set  of  players  in 
the  operating  environment  that  has  direct  interaction  with  the  system  under  test  and  culling  that 
group  down  to  a  minimum  set.  The  historical  approach  resulted  in  a  consistent  collection  of  test 
limitations  that  are  found  in  test  report  after  test  report,  e.g.,  insufficient  numbers  and  types  of 
targets;  insufficient  numbers  and  types  of  friendly  players;  and  inadequate  representation  of 
command,  control,  communications,  intelligence,  surveillance,  and  reconnaissance  assets  on 
friendly  and  opposing  sides. 

The  advantage,  at  the  cost  of  test  limitations,  of  the  historical  approach  to  test  concept 
development  and  planning  was  that  uncontrolled  variables  were  kept  to  a  minimum,  and  cause 
and  effect  relationships  were  relatively  straightforward.  This  was  not  a  trivial  advantage,  but  it 
was  an  advantage  for  the  analysts,  not  the  system  under  test  (SUT)  users.  In  combat  operations 
the  users  sometimes  found  that  factors  not  included  in  testing  had  significant  bearing  on  the 
ability  of  a  system  to  do  its  intended  job. 

It  is  a  plausible  working  assumption  that  ADS  can  technically  support  representation  of  the 
military  operating  environment  at  the  campaign  or  theater  level.  If  ADS  is  included  in  the  test 
planning  tool  kit  from  the  outset,  it  is  possible  to  begin  the  test  concept  development  process  at 
the  top  rather  than  the  bottom.  (The  “top”  may  not  be  at  theater  level;  it  is  established  by  the 
relevant  operational  task  or  tasks.)  The  methodology  described  in  this  paper  is  a  top-level 
methodology.  It  is  an  approach  which  is  compatible  with  the  “strategy  to  task”  or  “mission  level 
evaluation”  philosophy.  It  is  also  a  methodology  for  test  concept  development  which 
incorporates  the  consideration  of  ADS  —  it  is  not  an  ADS  planning  methodology.  This 
methodology  is  designed  to  provide  insights  on  whether  to  use  ADS  and  where  in  a  test  program 
the  use  might  fit. 

The  advantage  of  a  top-down  approach  to  test  concept  development  is  that  the  whole  gamut  of 
interactions  is  available  for  consideration  even  if  many  of  those  interactions  are  assessed  as 
irrelevant  and  excluded  from  the  final  concept.  The  top-down  approach  doesn’t  require  that 
every  possible  interaction  be  included  in  the  test,  but  it  does  require  an  item  by  item  assessment 
of  each  interaction.  Decisions  to  exclude  interactions  are  conscious  decisions  not  default 
decisions  as  a  function  of  a  bottom-up  approach. 

Mission-  or  task-level  evaluation  is  explicitly  a  top-down  approach.  The  top  level,  for  test 
planning  purposes,  may  be  much  lower  than  campaign  or  theater.  Just  how  high  the  top  level  is. 
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is  a  function  of  the  task  being  evaluated.  Some  systems  may  have  little  or  no  interaction  beyond 
a  unit  boundary,  and  others  may  interact  closely  with  the  theater  and  campaign  levels.  In  the  case 
of  DT&E,  it  is  necessary  to  substitute  “specification  sets”  for  “tasks.”  The  substitution  should 
not  be  difficult.  While  there  may  be  evolutionary  changes  as  a  program  evolves,  the  operational 
tasks  expected  of  a  new  system  are  known  as  a  result  of  mission  needs  analysis  and  serve  as  the 
basis  for  initial  requirements  development.  It  shouldn’t  be  hard  to  map  certain  system 
specifications  to  a  specific  task.  The  methodology,  as  described,  should  be  useful  for  much 
DT&E.  The  player  sets  in  DT&E  scenarios  might  be  smaller  relative  to  the  OT&E  cases. 

NOTE:  A  top-down  approach  won’t  help  much  if  it  is  implemented  with  a  mind  set  fixed  on 
historical  limitations.  It’s  necessary  for  the  test  planners  to  understand  that  ADS  provides 
opportunities  which  weren’t  previously  feasible. 

In  order  to  lend  structure  to  the  methodology,  a  SUT  has  been  assumed  which  is  a  hypothetical 
air-ground  attack  aircraft.  Please  bear  in  mind  that  this  is  just  an  example  of  a  process.  Some  of 
the  terminology  may  be  dated  or  inaccurate,  but  the  logic  for  the  process  should  hold  up.  The 
reader  should  have  little  difficulty  tailoring  the  steps  of  the  process  to  update  the  terminology  or 
to  apply  them  to  other  kinds  of  systems.  The  logic  flow  for  the  initial  elements  of  the  concept 
development  methodology  is  shown  in  Figure  1 . 
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Obtain  system  descriptions: 
ORD,  RCM,  COIS,  STAR, 
CONOPS,  etc. 


Obtain  operational  task 
list  from  system  users 


Select  a  task 


Develop  task  specific: 
measures  of  effectiveness  (MOE) 
measures  of  performance  (MOP) 
data  elements 


Develop  task  specific  operating 
environment 

friendly  forces,  objectives,  conditions, 
threat  forces,  interactions,  player  list 


Develop  operational  scenario 
and  player  list 


© 

/  Adequate  test\ 

/  scenario  with  \  '  e? 

\  traditional  / 

\  approaches?/ 

Create  traditional  test 
plan  for  selected  task 


Select  player 
(Begin  with 


Determine 

available 

forms 


Constructive 


elect  player  form: 
fidelity,  validity,  accessibility,  compatibility, 
cost 


N  /  Adequate  test\ 

/  scenario  with  ' 
\  ADS  augmented 
\  approaches?  / 


Create  ADS  test  plan 
for  selected  task 


Additional  tasks? 


ORD  -  operational  requirements 
document 

RCM  =  Requirements  Correlation  Matrix 
COI  -  critical  operational  issue 
STAR  =  system  threat  assessment 
report 

CONOPS  =  concept  of  operations 


Master  Test  Plan  1  Done 


Task  based  test  plan 


Figure  1.  Logic  Flow  for  Initial  Elements  of  the  Planning  Methodology 
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2.1  Step  1.  Understanding  the  System  Under  Test 

Step  1  requires  the  test  planners  to  research  the  acquisition  documentation  to  gain  a  thorough 
understanding  of  the  SUT  and  its  intended  operating  environment.  This  understanding 
incorporates  the  operational  tasks  the  system  is  designed  to  perform,  the  critical  system 
parameters,  the  system  and  operational  requirements,  the  concept  of  operations,  the  logistical 
support  concept,  and  the  top  level  or  general  operating  environment.  One  piece  of  the 
understanding  deals  with  the  technical  or  specification  aspects  of  the  system.  The  other  piece 
deals  with  the  interactions  between  the  technical  characteristics  of  the  system  and  the  world  it 
operates  in  from  a  strategic  perspective  —  the  friendly  and  supporting  forces,  the  natural 
environment,  and  the  threats  posed  by  the  enemy.  In  the  case  of  a  hypothetical  close  air  support 
(CAS)  aircraft,  the  cast  of  potential  players  from  a  theater  perspective  is  pretty  extensive. 

•  Joint  force  headquarters 

•  Tactical  air  operations  center  (TAOC) 

-  Air  task  orders 

-  Fragmentary  orders 

•  Wing  operations  centers  (WOCs) 

•  Tactical  operations  centers  (corps,  division,  brigade,  etc.) 

•  Theater  air  defense  structures 

-  Air  defense  units 

-  Control  centers 

-  Safe  passage  procedures 

•  Tactical  air  control  parties 

•  Forward  air  control  posts 

•  Airborne  battlefield  command  and  control  centers  (ABCCCs) 

•  Forward  air  control lers-air  (FAC-A) 

•  Forward  observers  (FO) 

•  Air  and  naval  gunfire  liaison  companies  (ANGLICOs) 

•  Direct/indirect  support  units  (suppressive  fires) 

•  Logistical  support  agencies/units 

-  Munitions 

-  Consumables 

•  Air  support  assets 

-  Combat  Air  Patrol 

-  Escort 

-  Suppression 

-  Refueling 

-  Combat  search  and  rescue 

•  Airborne  Warning  And  Control  System  (AW ACS) 

•  Joint  Surveillance  Target  Attack  Radar  System  (Joint  STARS) 

•  Airborne  laser  (ABL) 
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•  Rivet  Joint,  Guardrail,  and  other  stuff 

•  Allied  forces 

•  Threats 

-  Airborne  interceptors 

-  Surface-to-air  missiles  (SAM) 

-  Anti-aircraft  artillery  (AAA) 

-  Small  arms 

-  Ground  forces 

-  Airborne 

-  Special  forces 

-  Naval  forces 

-  At  sea 

-  Ashore 
-  Targets 

-  Troop  units 

-  Vehicles 

-  Fixed  targets 

-  Weapon  emplacements 

Obviously  not  all  the  listed  players  play  in  every  task  the  SUT  is  required  to  perform;  as  the  task 
changes,  the  relevant  players  list  changes. 

Technical  measures  are  straightforward  and  map  directly  to  system  requirements  and 
specifications.  The  technical  measures  tend  to  remain  constant  through  the  range  of  operational 
tasks,  although  some  of  them  may  be  stressed  in  some  tasks  and  not  in  others. 

Operational  measures  map  to  the  ability  of  a  system  to  perform  its  intended  tasks.  The 
environment  and  the  player  set  may  change  significantly  between  tasks  (e.g.,  day  to  night  or 
interdiction  to  CAS).  The  operational  measures  may  vary  significantly  from  task  to  task. 

Once  the  test  planners  have  a  thorough  understanding  of  the  system,  its  tasks,  its  operating 
environment  and  its  interactions  in  that  environment,  they  can  proceed  to  the  next  step. 

2.2  Step  2.  Select  a  Task 

This  step  involves  the  selection  of  a  specific  task.  A  complex  system  may  be  assigned  many 
operational  tasks.  Some  tasks  may  be  very  similar,  while  others  may  be  vastly  different.  It  is 
possible  that  similar  tasks  may  be  grouped  for  evaluation  purposes  and  tested  on  the  basis  of  a 
single  task.  In  our  hypothetical  case  we  might  choose  a  task  of  CAS  in  daylight  and  good 
weather  with  communications  jamming  (comjam),  and  a  moderate  threat  environment. 
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2.3  Step  3.  Develop  Relevant  Measures 

Once  a  specific  task  is  selected,  the  planners  can  develop  relevant  measures  for  the  task  and  a 
task-specific  operating  environment.  The  operating  environment  in  combination  with  assigned 
objectives  and  missions  provides  a  context  for  the  test  measures  and  defines  the  cast  of  players. 
Let’s  assume  for  our  illustrative  case  that  we  have  measures  associated  with  the  following  areas 
of  SUT  performance. 

•  Ability  of  the  pilot  in  the  CAS  aircraft  to  communicate  effectively  with  controlling, 
coordinating  and  supporting  agencies  in  a  comjam  environment 

•  Survivability 

•  Ability  to  navigate  to,  identify,  and  attack  the  appropriate  target 

•  Circular  error  probable  (CEP),  circular  error  average  (CEA) 

•  Weapons  effects 

Given  those  kinds  of  measures,  the  relevant  task  player  list  might  look  something  like  the 
following. 

•  System  under  test 

•  Tactical  air  operations  center 

-  Air  task  order 

-  Fragmentary  order 

•  Wing  operations  center 

•  Corps  tactical  operations  center  (CTOC) 

•  Theater  air  defense  structure 

-  Control  center 

-  Safe  passage  procedure 

•  Airborne  battlefield  command  and  control  centers 

•  Airborne  forward  air  controller  (FAC) 

•  Forward  observers  (FO) 

•  Air  support  assets 

-  Suppression 

-  Refueling 

•  Airborne  Warning  And  Control  System 

•  Joint  Surveillance  Target  Attack  Radar  System 

•  Allied  forces 

•  Threats 

-  Airborne  interceptors 

-  Surface-to-air  missiles 

-  Anti-aircraft  artillery 

-  Small  arms 
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•  Targets 

-  Troop  units 

-  Vehicles 

In  order  to  structure  a  test,  the  player  cast  has  to  be  embedded  in  a  dynamic  operational  scenario. 
The  scenario  supports  detailed  mission  layout  activities  and  time-sequenced  events  for  the  SUT. 
Some  of  the  players  on  the  list  have  very  tightly  coupled  interactions  with  the  SUT.  Examples 
include  friendly  ground  forces,  enemy  ground  forces  (targets),  terminal  threats,  and  the  airborne 
FAC.  Other  agencies  such  as  AWACS,  Joint  STARS,  or  the  WOC  have  more  loosely  coupled 
interactions  but  may  be  important  to  SUT  performance. 

Step  3  is  finished  when  the  cast  of  players  has  been  whittled  down  to  those  who  play  a  role  in  the 
performance  of  the  SUT  in  the  selected  task,  and  the  measures  of  performance  (MOPs)  and 
measures  of  effectiveness  (MOEs)  have  been  developed  for  the  SUT  evaluation.  The  scenario 
developed  in  Step  3  is  an  operational  scenario;  a  real  world  scenario  --  not  a  test  scenario.  That 
comes  later. 

2.4  Step  4.  Consider  Using  ADS 

The  activity  described  to  this  point  is  simply  a  test  planning  or  test  concept  development 
approach.  Step  4  is  a  switch  point  -  to  include  or  not  to  include  a  detailed  consideration  of  ADS 
use  as  part  of  the  concept  development  methodology.  In  a  few  cases,  the  operating  and  test 
environments  may  be  relatively  simple.  An  example  might  be  the  testing  of  a  new  side  arm.  A 
test  of  such  a  system  can  be  done  live,  in  open  air,  and  affordably.  The  example  I  have  chosen  is 
complex  and  highly  interactive  with  a  wide  cast  of  players.  In  the  complex  case,  it  is  reasonably 
clear  that  live,  open  air  testing  with  the  full  cast  of  players  is  not  affordable.  Because  of  the  large 
number  of  man-in-the-loop  (MITL)  interactions,  it  is  also  reasonably  clear  that  representation  of 
the  environment  in  a  stand-alone  simulation  will  suffer  credibility  problems  because  we  don’t 
model  human  decision  making  very  well.  If  the  test  planners  are  reasonably  certain  that  the  test 
environment  cannot  be  adequately  represented  using  the  traditional  test  approaches,  then  they 
have  two  choices:  accept  the  test  limitations  or  explore  ADS  to  see  if  the  technology  can  make  a 
better  test  within  the  fiscal  constraints. 

Stated  succinctly,  the  decision  associated  with  Step  4  is  about  the  adequacy  of  the  test  scenario  as 
compared  with  the  operational  scenario.  In  a  world  with  no  fiscal  or  safety  constraints,  the 
operational  scenario  and  the  test  scenario  would  be  identical.  In  the  real  world,  the  issue 
becomes  “can  we  approximate  reality  with  sufficient  accuracy  to  have  a  satisfactory  test.”  If  the 
test  planners  cannot  provide  an  appropriate  test  environment  using  traditional  test  approaches, 
then  the  answer  is  “no,”  and  the  planners  should  explore  whether  ADS  can  do  anything  to 
improve  the  situation.  If  the  answer  is  “no,”  then  the  process  moves  along  to  Step  5.  If  the 
answer  is  “yes,  we  can  provide  a  suitable  test  environment,”  then  the  planners  can  proceed  with  a 
traditional  test  plan  for  that  particular  operational  task.  Other  tasks  may  require  different 
approaches. 
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2.5  Step  5.  Select  a  Player 


Traditional  testing  shortfalls  often  include  an  insufficient  number  of  test  articles,  insufficient 
number  of  threats,  and  inadequate  representation  of  friendly  force  interactions.  The  process  of 
ADS  exploration  begins  with  a  visit  to  the  player  list  developed  in  Step  3,  and  the  first  player  on 
that  list  is  the  SUT.  Depending  on  where  the  program  is,  the  SUT  may  be  available  in  a  variety 
of  forms.  Early  in  the  program,  the  SUT  may  only  be  available  as  a  digital  system  model  (DSM). 
Later  the  SUT  may  exist  in  brassboard  form  with  a  variety  of  subcomponents  scattered  among  a 
variety  of  vendors.  Eventually  the  SUT  will  be  available  in  prototype  or  production  version 
form,  and  a  flight  simulator  version  will  emerge.  (The  DSM  version  is  still  available  at  this 
stage.) 

2.6  Step  6.  Determine  Player  Representation 

The  determination  of  a  player  representation  will  be  made  based  on  both  the  availability  of 
representations  and  the  test  objectives.  This  step  addresses  both  of  these  criteria.  The  planning 
team  will  need  to  investigate  the  available  representations  for  each  player  beginning  with  the 
SUT.  The  planning  team  will  also  have  to  define  the  requirements  for  the  test  to  meet  the  test 
objectives. 

Let’s  assume  that  we  have  a  prototype  version  of  the  SUT  and  a  flight  simulator  exists.  We  also 
have  a  DSM  form  of  the  SUT.  If  the  operational  doctrine  calls  for  a  two-ship  flight,  then  we 
have  a  requirement  to  represent  two  SUTs.  One  approach  could  be  to  couple  the  SUT  manned 
flight  simulator  with  another  manned  simulator  that  could  represent  the  SUT  and  the 
communications  types  and  channels  for  interflight  communications.  A  facility  like  the  Theater 
Air  Command  and  Control  Simulation  Facility  (TACCSF)  at  Kirtland  Air  Force  Base,  New 
Mexico,  could  represent  a  second  version  of  the  SUT  at  a  reasonable  cost.  Most  manned  dome 
simulators  could  provide  the  visual  representation  of  the  TACCSF-generated  wingman.  The 
technology  exists  to  input  or  overlay  jamming  energy  on  the  communications  link. 

After  looking  at  the  SUT  configuration  choices,  the  planners  can  move  on  to  the  other  players.  If 
the  player  list  is  prioritized  on  the  basis  of  the  more  direct  interactions,  then  the  more  important 
players  are  addressed  first.  For  each  player,  the  test  planners  must  have  access  to  information 
about  the  various  manifestations  of  the  player.  They  must  know  what  forms  are  available,  and 
they  must  learn  what  they  can  about  capabilities  and  costs  for  each  form. 

A  facility  like  TACCSF  could  represent  the  airborne  FAC  complete  with  man-in-the-loop.  If 
there  was  a  higher  fidelity  requirement,  a  manned  simulator  of  the  FAC  aircraft  could  be  used. 
The  communications  links  could  be  subjected  to  jamming  in  the  same  manner  as  the  interflight 
links. 

The  targets  and  the  friendly  ground  forces  in  the  vicinity  of  the  strike  also  have  tight  interactions 
with  the  SUT.  Individual  targets  can  be  simulated  in  large  numbers  by  a  number  of  constructive 
simulations  --  Janus  serves  as  an  example.  Target  visual  representations  can  be  piped  to  the 
simulator  dome  for  the  SUT.  Friendly  command  and  control  centers  of  all  types  can  be 
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represented  at  a  level  of  fidelity  tailored  to  test  needs  and  cost.  A  corps  tactical  operations  center 
(TOC),  for  instance,  could  be  represented  by  a  full-up,  fielded  TOC  if  testing  was  done  in 
conjunction  with  some  exercise.  The  TOC  could  also  be  represented,  at  decreasing  levels  of 
fidelity  and  cost,  as  a  partially  manned  TOC  in  garrison;  as  an  emulation  of  a  TOC  in  TACCSF 
or  a  similar  simulation  facility;  or  by  a  single  individual  reading  from  a  script.  The  same 
approach  can  be  applied  to  other  command  and  control  elements  such  as  the  tactical  air  control 

party. 

Another  tight  interaction  involves  the  SUT  and  the  threat  systems  in  the  strike  area.  The  threats 
can  be  represented  at  any  level  from  constructive  to  live.  If  high  fidelity  is  a  requirement  (and  in 
our  hypothetical  example  it  is)  and  since  survivability  is  a  measure  of  interest,  then  hardware-in- 
the-loop  may  be  a  minimum  requirement. 

As  the  players  (the  air  operations  center  (AOC)  and  the  WOC,  for  instance  )  become  more  and 
more  peripheral,  fidelity  requirements  decrease  and  scripted  inputs  or  constructive  models  will 

suffice. 

Step  6  involves  a  lot  of  research  and  learning. 

2.7  Step  7.  Fidelity  Versus  Cost 

Step  7  involves  the  art  of  compromise  between  fidelity  and  cost.  With  the  information  gleaned 
in  Step  6,  the  test  planners  are  in  a  position  to  make  reasoned  choices  about  the  players  in  the  test 
and  the  appropriate  form  of  representation  for  each  of  them.  Rough  order  of  magnitude  (ROM) 
costs  are  adequate  in  the  test  concept  development  process.  Costs  are  refined  in  detailed  test 
planning. 

2.8  Step  8.  Evaluate  Adequacy  of  the  Environment 

When  the  process  reaches  Step  8,  the  question  facing  the  planners  involves  the  adequacy  of  the 
environment  which  will  be  created  by  the  interactions  of  the  players  selected.  Each  time  Steps  5 
through  7  are  executed,  the  planners  must  ask  whether  more  players  are  needed.  If  so,  they  return 
to  Step  5.  Step  5  is  executed  repeatedly  until  the  test  planners  are  satisfied  that  the  test 
environment  is  rich  enough  in  terms  of  meaningful  interactions  to  support  a  sound  test. 

2.9  Step  9.  Initial  Planning 

When  the  test  planners  are  satisfied  with  the  test  concept  for  a  given  operational  task,  they  can 
proceed  with  initial  test  planning  for  the  associated  test.  The  initial  planning  is  conducted  to  the 
level  required  by  Step  11. 
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2.10  Step  10.  Additional  Tasks 


Step  10  involves  the  examination  of  the  functionalities  of  the  SUT,  and  the  assessment  of  the 
necessity  for  further  testing  for  additional  operational  tasks.  If  there  are  additional  operational 
tasks  that  differ  enough  from  those  already  addressed  in  test  planning  to  this  point,  then  the 
planners  need  to  loop  back  to  Step  2  of  this  process  and  develop  another  test  concept.  Planning 
cannot  move  on  to  Step  1 1  until  initial  planning  has  been  completed  for  all  operational  tasks. 

2.11  Step  11.  Develop  Master  Plan 

The  task  associated  with  Step  1 1  involves  the  deconfliction  and  coordination  of  each  of  the 
individual  task-oriented  test  segments.  Essentially  Step  11  involves  the  development  of  the 
master  plan  and  schedule. 

In  summary,  this  is  a  top-level  test  concept  development  process  which  incorporates  decision 
elements  associated  with  the  use  of  ADS  for  test  and  evaluation.  It  is  only  intended  as  an 
example,  and  each  test  program  is  going  to  have  unique  aspects  which  will  require  planners  to 
use  creativity  and  imagination.  Hopefully  this  outline  approach  provides  the  test  planners  with 
an  understanding  that  ADS  offers  test  opportunities  that  were  never  there  before.  ADS  can  be 
used  to  good  effect  at  reasonable  cost  if  it  is  used  intelligently.  There  are  many  things  ADS 
won’t  remedy,  but  there  are  many  things  it  will.  ADS  is  a  technology  set  which  should  be 
considered  during  test  planning.  If  it  doesn’t  make  sense  to  use  it,  don’t  use  it.  On  the  other 
hand,  don’t  dismiss  it  just  because  you  think  it’s  too  hard,  or  too  exotic,  or  too  expensive. 
Detailed  test  planning  approaches  are  discussed  in  the  test  planning  methodology  section  which 
follows. 
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3.0  ADS-Based  Test  Planning  and  Implementation  Methodology 

Assuming  the  decision  is  made  to  implement  ADS-based  testing,  the  following  methodology 
applies.  This  methodology  follows  the  steps  given  in  the  HLA  FEDEP  model  [Ref.  1].  In 
comparing  these  guidelines  with  the  FEDEP  model,  note  that  the  terms  “ADS  architecture”  and 
“distributed  test”  used  here  equate  to  the  term  “federation”  in  the  FEDEP  model,  and  the  terms 
“facilities,”  “participants,”  and  “players”  used  here  equate  to  the  term  “federates”  in  the  FEDEP 
model. 

The  FEDEP  model  groups  the  activities  needed  to  develop  and  execute  a  distributed  test  into  six 
steps. 

Step  1:  The  test  sponsor  or  evaluator  and  the  distributed  test  development  team  define  and  agree 
on  a  set  of  objectives  and  document  what  must  be  accomplished  to  achieve  those  objectives. 

This  is  a  test  planning  step  and  is  addressed  by  the  test  planning  methodology. 

Step  2:  A  representation  of  the  real-world  domain  of  interest  is  developed  and  described  in 
terms  of  a  set  of  required  objects  and  interactions.  Most  of  the  activities  under  this  step  are 
addressed  by  the  test  planning  methodology. 

Step  3:  Distributed  test  participants  (federates)  are  determined,  and  required  functionalities  are 
allocated  to  the  participants. 

Step  4:  The  federation  object  model  (FOM)  is  developed  (if  HLA  is  implemented),  participant 
agreements  on  consistent  databases/algorithms  are  established,  and  modifications  to  federates  are 
implemented  (as  required). 

Step  5:  All  necessary  distributed  test  implementation  activities  are  performed,  and  testing  is 
conducted  to  ensure  interoperability  requirements  are  being  met. 

Step  6:  The  distributed  test  is  executed,  outputs  are  generated,  and  results  provided. 

The  FEDEP  model  breaks  the  six  steps  into  activities,  as  shown  in  Table  1.  The  following 
subsections  describe  the  activities  for  each  step  in  more  detail.  Where  appropriate,  lessons 
learned  from  Joint  Advanced  Distributed  Simulation  (JADS)  testing  experience  are  given.  Also, 
the  cost  impact  factors  applying  to  selected  steps  are  discussed  in  Appendix  A,  Cost  Factors  For 
ADS-Based  Testing,  which  lists  the  15  top  cost  factors  in  order  of  decreasing  impact.  In 
addition.  Appendix  B,  Guide  to  the  VV&A  of  an  ADS-Enhanced  Test  Environment,  discusses 
verification,  validation,  and  accreditation  (VY&A)  considerations  which  apply  to  each  FEDEP 
step. 
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Table  1.  Distributed  Test  Planning  and  Implementation  Activities 


STEP 

ACTIVITIES 

1.  Define  Distributed  Test  Objectives 

1 . 1  Identify  Needs 

1.2  Develop  Objectives 

2.  Develop  Conceptual  Model 

2. 1  Develop  Scenario 

2.2  Perform  Conceptual  Analysis 

2.3  Develop  Test  Requirements 

3.  Design  Distributed  Test 

3.1  Select  Participants 

3.2  Allocate  Functionality 

3.3  Prepare  Plan 

4.  Develop  Distributed  Test 

4.1  Develop  FOM 

4.2  Establish  Participant  Agreements 

4.3  Implement  Participant  Modifications 

5.  Integrate  and  Test  Architecture 

5.1  Plan  Execution 

5.2  Integrate  and  Test  ADS  Architecture 

6.  Execute  Distributed  Test  and  Analyze 
Results 

6. 1  Execute  Distributed  Test 

6.2  Process  Output 

6.3  Prepare  Results 

Although  the  FEDEP  steps  are  presented  in  a  sequential  fashion,  experience  has  shown  that  many 
of  the  activities  in  Table  1  are  actually  cyclic  with  extensive  feedback  among  activities  and/or 
concurrent.  Implementers  should  not  enforce  a  strict  waterfall  approach  to  the  steps  given.  Not 
only  may  variations  in  the  order  of  the  activities  be  appropriate,  but  it  is  frequently  necessary  to 
revisit  previous  activities  as  the  distributed  test  requirements  and  design  become  more  mature. 

3.1  Step  1.  Define  Distributed  Test  Objectives 

According  to  the  FEDEP  model,  the  purpose  of  the  activities  for  this  step  is  to  define  and 
document  a  set  of  needs  that  are  to  be  addressed  through  the  development  and  execution  of  a 
distributed  test  and  to  transform  these  needs  into  a  more  detailed  list  of  specific  test  objectives. 
The  key  activities  for  this  step  and  the  activity  inputs  and  outputs  are  shown  in  Figure  2  [Ref.  1]. 
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Figure  2.  Define  Distributed  Test  Objectives 
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JADS  experience  has  shown  the  importance  of  forming  an  integrated  product  team  (IPT)  early  in 
the  planning  stage  of  a  distributed  test.  A  distributed  test  usually  brings  together  organizations 
unfamiliar  with  conducting  coordinated,  distributed  T&E  tests.  The  linking  procedures, 
communications,  test  control,  and  security  aspects  are  different  from  stand-alone  facility  tests. 
IPT  meetings  help  ensure  a  common  understanding  of  these  unique  aspects.  Minutes  from  the 
meetings  should  be  kept  and  action  items  tracked  throughout  the  test  development  effort  in  order 
to  provide  a  more  structured  and  cohesive  approach  and  to  assist  in  resolving  issues.  An  efficient 
method  for  conducting  IPT  meetings  is  by  use  of  secure  video  teleconferences  (VTCs).  VTCs 
reduce  the  need  for  key  personnel  to  travel  to  a  common  location  and  result  in  less  time  away 
from  the  job  of  preparing  for  the  distributed  test. 

A  key  member  of  the  IPT  is  the  system  integrator.  The  system  integrator  is  needed  to  oversee  the 
distributed  test  development  by  coordinating  the  integration  of  the  various  organizations/facilities 
involved.  The  system  integrator  must  be  able  to  grasp  the  technical  and  administrative  issues 
involved  and  must  have  the  authority  to  proactively  resolve  any  problems  (cost  impact  factor  of 
rank  #7  -  see  Appendix  A). 

3.1.1  Activity  1.1  -  Identify  Needs 

According  to  the  FEDEP  model,  the  primary  purpose  of  this  activity  is  to  develop  a  clear 
understanding  of  the  problem  to  be  addressed  by  the  distributed  test.  Inputs  to  this  activity  are 
the  program  objectives  and  information  on  resources  available  to  support  a  distributed  test.  The 
main  output  of  this  activity  is  a  needs  statement  which  includes  the  following 

•  High-level  descriptions  of  critical  systems  of  interest 

•  Coarse  indications  of  fidelity  and  required  behaviors  for  simulated  players 

•  Key  events  that  must  be  represented  in  the  distributed  test  scenario 

•  Output  data  requirements 

•  Resources  that  will  be  available  to  support  the  distributed  test  (e.g.,  funding,  personnel,  tools, 
facilities) 

•  Any  known  constraints  which  may  affect  how  the  distributed  test  is  developed  (e.g.,  due 
dates,  security  requirements) 

Note  that  most  of  these  items  should  have  been  determined  during  application  of  the  test  concept 
development  methodology.  However,  their  determination  should  be  repeated  here  to 
check/validate  the  earlier  findings. 

3.1.2  Activity  1.2  -  Develop  Objectives 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  refine  the  needs  statement  into  a 
more  detailed  set  of  specific  objectives  for  the  distributed  test.  This  activity  requires  close 
collaboration  between  the  distributed  test  user/sponsor  and  the  test  development  team  to  ensure 
that  the  resulting  objectives  meet  the  stated  needs.  The  user/sponsor  must  clearly  define, 
communicate,  and  document  test  requirements  early  in  the  test  planning  phase.  The  main  input 
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to  this  activity  is  the  needs  statement  from  the  previous  activity.  The  main  outputs  of  this 
activity  are  a  statement  of  the  test  objectives  and  initial  planning  documents.  The  test  objectives 
statement  should  include  the  following  information 

•  A  prioritized  list  of  measurable  test  objectives 

•  A  high-level  description  of  key  ADS  architecture  characteristics  (e.g.,  repeatability, 
portability,  time  management  approach) 

•  Needed  equipment,  facilities,  and  data 

•  Operational  context  constraints  or  preferences,  including  friendly/threat/civilian  order  of 
battle,  geographic  regions,  environmental  conditions,  and  tactics 

•  Identification  of  security  position,  including  estimated  security  level  and  possible  designated 
approval  authority(s) 

•  A  configuration  management  approach 

•  Identification  of  tools  to  support  scenario  development,  conceptual  analysis,  VV&A  (see 
Appendix  B  for  VV&A  activities  to  consider)  and  test  activities,  and  configuration 
management 

The  initial  test  planning  documents  should  include  a  test  development  plan  with  an  approximate 
schedule  and  major  milestones  and  initial  versions  of  the  test  plan,  accreditation  plan,  and 
security  plan  (cost  impact  factor  of  rank  #12  -  see  Appendix  A). 

Note  that  most  of  these  items  should  have  been  determined  during  application  of  the  test  concept 
development  methodology.  However,  their  determination  should  be  repeated  here  to 
check/validate  the  earlier  findings. 

3.2  Step  2.  Develop  Conceptual  Model 

According  to  the  FEDEP  model,  the  purpose  of  this  step  is  to  develop  an  appropriate 
representation  of  the  real-world  domain  that  applies  to  the  distributed  test  environment  and  to 
develop  the  test  scenario.  During  this  step,  test  objectives  are  transformed  into  a  set  of  specific 
requirements  for  use  as  success  criteria  during  ADS  architecture  testing.  The  key  activities  for 
this  step  and  the  activity  inputs  and  outputs  are  shown  in  Figure  3  [Ref.  1]. 
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Figure  3.  Develop  Conceptual  Model 

3.2.1  Activity  2.1  -  Develop  Scenario 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  develop  a  functional 
specification  of  the  test  scenario.  The  primary  input  to  this  activity  is  the  operational  context 
constraints  specified  in  the  test  objectives  statement,  although  existing  scenario  databases  may 
also  provide  a  reusable  starting  point  for  scenario  development.  The  primary  output  is  the  test 
scenario.  The  scenario  description  should  include  the  following 

•  The  types  and  numbers  of  major  players  that  must  be  represented  in  the  distributed  test 

•  A  functional  description  of  the  capabilities,  behavior,  and  relationships  among  these  major 
players  over  time 

•  A  specification  of  relevant  environmental  conditions  that  impact  or  are  impacted  by  players 
in  the  distributed  test 

•  Initial/terminal  conditions  and  the  specific  map  projection  chosen  for  the  scenario 

Note  that  most  of  these  items  should  have  been  determined/developed  during  application  of  the 
test  concept  development  methodology.  However,  their  determination/development  should  be 
repeated  here  to  check/validate  the  earlier  findings. 

3.2.2  Activity  2.2  -  Perform  Conceptual  Analysis 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  produce  a  conceptual  model  of 
the  ADS  environment.  The  primary  inputs  to  this  activity  are  the  test  scenario  from  the  previous 
activity,  the  test  objectives  statement,  and  any  doctrine  and  tactics  appropriate  for  the  scenario. 
The  output  of  this  activity  is  the  conceptual  model  which  provides  an  implementation- 
independent  representation  that  serves  as  a  vehicle  for  transforming  objectives  into  functional 
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and  behavioral  capabilities,  and  provides  a  crucial  traceability  link  between  the  test  objectives 
and  the  design  implementation.  The  conceptual  model  is  a  description  of  the  players,  their 
actions,  and  any  interactions  among  players  that  need  to  be  included  in  the  distributed  test  in 
order  to  achieve  all  test  objectives.  These  are  described  without  any  reference  to  specific 
simulations  that  will  be  used. 

Once  this  is  completed,  the  conceptual  model  needs  to  be  carefully  evaluated  before  the  next 
activity.  Revisions  to  the  original  federation  objectives  may  be  defined  and  implemented  as  a 
result  of  this  feedback. 

Note  that  the  conceptual  model  description  should  have  been  developed  during  application  of  the 
test  concept  development  methodology.  However,  this  development  should  be  repeated  here  to 
check/validate  the  earlier  findings. 

3.2.3  Activity  2.3  -  Develop  Test  Requirements 

According  to  the  FEDEP  model,  the  conceptual  model  will  lead  to  the  definition  of  detailed 
distributed  test  requirements  and  test  evaluation  criteria.  These  requirements  should  be  based  on 
the  distributed  test  objectives,  should  be  directly  testable,  and  should  provide  the 
implementation-level  guidance  needed  to  design  and  develop  the  distributed  test  (cost  impact 
factor  of  rank  #9  -  see  Appendix  A).  The  test  requirements  will  also  be  the  basis  for  the  criteria 
for  evaluating  test  results  (see  Fig.  3).  Major  top-level  requirements  which  should  be  addressed 
include  the  following  (although  some  of  these  requirements  should  have  been  developed  during 
application  of  the  test  planning  methodology,  their  development  should  be  repeated  here  to 
check/validate  the  earlier  findings). 

-  Fidelity  requirements. 

-  The  fidelity  requirements  for  all  players  represented  in  the  distributed  test  scenarios  must 
be  determined.  The  required  fidelity  depends  upon  the  maturity  of  the  SUT,  the  SUT  test 
objectives,  and  the  nature  of  the  interactions  between  the  SUT  and  the  other  players. 

-  The  fidelity  of  the  SUT  representation  may  be  limited  to  available  models  or  test  articles. 
For  example,  during  early  DT&E,  a  low-fidelity  digital  model  may  be  the  only  SUT 
representation  available,  but  during  late  DT&E  and  OT&E,  possible  SUT  representations 
may  include  high-fidelity  digital  models,  hardware-in-the-loop  (HWIL)  labs,  and  live  test 
articles.  If  multiple  SUT  representations  are  available  with  varying  levels  of  fidelity,  the 
choice  will  usually  be  driven  by  the  SUT  test  objectives  and  other  considerations  such  as 
availability  and  cost. 

-  The  required  fidelity  for  the  other  players  normally  depends  on  the  fidelity  of  the  SUT, 
the  sensitivity  of  test  objectives/measures  to  player  interactions  with  the  SUT,  the 
strength  of  the  interactions  with  the  SUT  (players  that  have  strong,  or  tightly  coupled, 
interactions  with  the  SUT  will  generally  have  higher  fidelity  requirements  than  those 
which  do  not),  the  test  objectives,  and  cost  and  availability  considerations. 

-  It  is  important  to  involve  SUT  experts  from  the  beginning  of  the  distributed  test  program 
in  order  to  determine  fidelity  requirements,  establish  the  data  and  instrumentation 
requirements,  verify/validate  the  analytical  approach,  assist  in  the  development  of  test 
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matrices  and  test  procedures,  and  provide  overall  SUT  expertise.  The  support  of  more 
than  one  SUT  expert  should  be  planned  for  (and  budgeted  for)  throughout  the  test. 

Interaction  requirements. 

-  Use  the  conceptual  model  to  determine  the  data  types  that  must  be  exchanged  among 
players  to  permit  interactions,  including  entity  state  data,  tactical  messages,  launch  and 
detonation  indications  (if  appropriate),  and  trial  start  and  stop  notification. 

Latency  requirements  (cost  impact  factor  of  rank  #4  -  see  Appendix  A). 

-  Determine  the  maximum  acceptable  latency  and  latency  variations  for  each  pair  of 
interacting  players.  The  maximum  latency  requirement  will  be  determined  by  how 
closely  coupled  the  interactions  are  and  by  the  maximum  allowable  error  in  the  location 
of  one  player  as  perceived  by  the  other. 

—  For  example,  if  the  two  players  are  reacting  to  each  other  in  real  time  (i.e.,  a  tightly- 
coupled,  closed-loop  interaction),  the  maximum  allowable  latency  will  be  relatively 
smaller  in  order  to  achieve  an  acceptable  error  in  the  perceived  locations  of  the 
players.  (An  estimate  of  the  position  error  for  one  player  as  perceived  by  the  other  is 
given  by  the  latency  of  the  entity  state  data  multiplied  by  the  first  player’s  velocity; 
hence,  an  estimate  of  the  maximum  allowable  latency  would  be  given  by  the 
maximum  acceptable  position  error  divided  by  the  first  player’s  velocity.) 

—  An  air-to-air  missile  guiding  to  intercept  an  aircraft  target  which  is,  in  turn, 
maneuvering  in  order  to  evade  the  missile  is  a  specific  example  of  a  tightly  coupled, 
closed-loop  interaction.  In  this  case,  the  maximum  acceptable  position  error  might  be 
the  lethal  radius  of  the  missile,  so  that  the  target  and  missile  do  not  disagree  on 
whether  the  target  was  killed. 

—  For  human-in-the-loop  interactions  (e.g.,  two  pilots  reacting  to  each  other),  the  rule  of 
thumb  for  the  maximum  acceptable  latency  is  about  100  milliseconds. 

—  For  uncoupled,  or  open-loop,  interactions,  there  may  be  no  maximum  acceptable 
latency  requirement.  An  air-to-air  missile  guiding  to  intercept  an  aircraft  target  which 
is  executing  a  scripted,  nonreactive  maneuver  is  a  specific  example  of  an  uncoupled, 
open-loop  interaction.  (This  is  the  typical  scenario  for  a  live  fire  test  against  an 
unmanned  drone  target.)  Another  specific  example  would  be  a  surface-to-surface 
missile  guiding  to  a  stationary  ground  target.  In  both  cases,  latency  would  not  affect 
the  outcome  of  the  missile  attack  as  perceived  by  either  player. 

—  Random  sample-to-sample  latency  variations  in  the  entity  state  data  received  over  a 
wide  area  network  (WAN)  have  been  observed  during  JADS  testing  and  result  in  an 
uncertainty  in  the  transmitting  player’s  position  as  perceived  by  the  receiving  player 
[Ref.  2].  An  estimate  of  the  position  uncertainty  is  given  by  the  standard  deviation  of 
the  latency  values  multiplied  by  the  transmitting  player’s  velocity.  Hence,  an  estimate 
of  the  maximum  allowable  latency  variations  would  be  given  by  the  maximum 
acceptable  position  uncertainty  divided  by  the  transmitting  player’s  velocity. 

Data  reliability  requirements. 

-  Determine  the  maximum  acceptable  level  of  ADS-induced  errors,  such  as  dropout  rate 
and  out-of-order  data  messages.  The  allowable  errors  may  vary  with  data  types.  For 
example,  some  loss  of  entity  state  data  may  be  tolerable  for  short  durations  if  dead 
reckoning  can  supply  the  missing  data  within  acceptable  error  tolerances.  However,  the 
loss  of  a  single  discrete  message  may  invalidate  an  entire  trial.  This  determination  may 
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drive  the  reliability  requirement  for  data  transport  and  guide  the  selection  of  data 
transport  protocols. 

-  Data  analysis  requirements. 

-  Draft  a  preliminary  data  management  and  analysis  plan  (DMAP)  that  details  the  analysis 
approach  for  each  test  objective.  From  the  DMAP  determine  which  data  must  be 
collected  and  the  analysis  techniques  to  be  applied. 

-  Identify  all  data  required  to  achieve  test  objectives.  This  generally  includes  all  data 
exchanged  among  players,  data  needed  for  test  control  and  monitoring,  and  data  which 
are  not  exchanged  but  are  needed  for  analysis.  An  example  of  the  latter  is  internal 
simulation  data  which  are  not  transmitted  to  other  players,  but  which  must  be 
analyzed/monitored  to  verify  proper  operation  of  the  simulation  during  linked  operations. 

-  Identify  all  data  required  to  analyze  network  performance  and  the  impact  of  network 
performance  on  the  players  (e.g.,  impact  on  SUT  data  validity). 

-  For  standard  analysis  techniques  (e.g.,  statistical  or  regression  analysis),  identify 
commercial  or  government  off-the-shelf  tools  that  can  be  used  for  data  analysis/reduction. 

-  Determine  the  requirements  for  any  custom  analysis/processing  tools. 

After  all  these  requirements  have  been  developed,  the  capability  of  the  support  agencies  (e.g., 
simulation  or  range  facilities,  networking  and  engineering  team)  to  support  the  test  must  be 
clearly  stated  and  documented,  such  as  by  a  statement  of  capability  (SOC).  The  SOC  documents 
the  set  of  requirements  and  provides  a  clear  statement  of  the  support  agency’s  capabilities, 
constraints,  and  limitations  in  meeting  those  requirements. 

The  support  agencies  also  need  to  create  an  integrated,  detailed  work  breakdown  structure  (WBS) 
early  in  the  program  which  is  consistent  with  the  SOC.  It  is  also  important  to  have  accurate  cost 
estimates  allocated  against  the  WBS  tasks  in  order  to  help  program  management  decisions. 

3.3  Step  3.  Design  Distributed  Test 

According  to  the  FEDEP  model,  the  purpose  of  this  step  is  to  identify,  evaluate,  and  select  all 
distributed  test  participants  (federates),  allocate  required  functionality  to  those  participants,  and 
develop  a  detailed  plan  for  test  bed  development  and  implementation.  The  key  activities  for  this 
step  and  the  activity  inputs  and  outputs  are  shown  in  Figure  4  [Ref.  1]. 
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Test  Requirements 


3.3.1  Activity  3.1  -  Select  Participants 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  determine  the  suitability  of 
individual  player  representations  (e.g.,  simulations,  HWIL  labs,  or  live  players/ranges)  to  become 
participants  in  the  distributed  test.  The  input  to  this  activity  is  the  conceptual  model  developed  in 
Activity  2.2.  The  output  is  an  identification  of  the  specific  player  representations  selected. 

This  activity  involves  the  identification  of  specific  simulations,  HWIL  labs,  or  live  players/ranges 
to  be  used  in  the  distributed  test  and  their  locations.  This  selection  is  driven  primarily  by  the 
following  factors: 

•  Perceived  ability  of  potential  representations  to  represent  the  players’  behavior  and  the 
interactions  specified  in  the  conceptual  model 

•  Fidelity  requirements  for  each  player 

•  Managerial  constraints,  such  as  availability  (cost  impact  factor  of  rank  #10  -  see  Appendix 
A),  cost,  schedule,  and  security  considerations 

•  Technical  constraints,  such  as  VV&A  status  and  portability 

•  For  live  players,  the  selection  of  particular  test  ranges  is  also  driven  by  considerations  of 
range  instrumentation  quality  and  quantity  and  data  processing  capability 

A  good  source  of  information  on  available  simulations  is  the  HLA  Object  Model  Library  (OML) 
which  contains  the  simulation  object  models  (SOMs)  describing  many  simulations.  The 
searching  and  browsing  features  provided  by  the  OML  expedite  the  search  of  the  electronic 
libraries  of  SOMs.  However,  it  is  highly  advisable  to  directly  verify  the  advertised  capabilities  of 
the  facilities/instrumentation  before  making  a  selection. 
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If  no  available  representations  meet  the  requirements  for  one  of  the  players,  either  the 
requirements  must  be  revised  or  a  new  representation  must  be  developed. 

3.3.2  Activity  3.2  -  Allocate  Functionality 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  allocate  the  responsibility  to 
represent  the  entities  and  actions  in  the  conceptual  model  to  the  participants.  This  activity  will 
allow  for  the  assessment  of  whether  the  set  of  selected  participants  provides  the  full  set  of 
required  functionality  or  whether  one  or  more  of  the  representations  will  need  to  be  enhanced  to 
meet  the  distributed  test  requirements.  The  inputs  to  this  activity  are  the  identified  participants 
from  the  previous  activity,  along  with  the  test  requirements,  the  test  scenario,  and  the  conceptual 
model.  The  output  is  allocated  requirements  for  the  participants,  including  any  requirements  for 
modifying  existing  player  representations  or  designing  new  ones. 

Requirements  need  to  be  allocated  to  the  participants  before  the  architecture  can  be  designed  and 
before  the  requirements  for  modifying  existing  player  representations  or  designing  new  ones  can 
be  determined.  These  allocated  requirements  include  the  following. 

-  Data  requirements. 

-  Determine  the  rates  of  data  exchanged  among  the  players.  For  each  pair  of  players, 
compare  the  rate  that  the  transmitting  player  generates  data  with  the  rate  that  the  receiving 
player  requires  inputs  (e.g.,  a  receiving  simulation’s  frame  rate). 

-  If  the  receiving  player  requires  data  at  a  lower  rate  than  generated  by  the  transmitting 
player,  determine  if  the  transmitting  player’s  output  can  be  filtered  down  to  the 
receiving  player’s  required  input  rate  before  the  transmitting  player’s  data  are 
interfaced  to  the  WAN  link  connecting  the  two. 

-  If  the  receiving  player  requires  entity  state  data  at  a  higher  rate  than  generated  by  the 
transmitting  player,  determine  if  the  received  data  can  be  dead  reckoned  at  the 
receiving  player’s  node  in  order  to  produce  the  required  input  rate.  This  is  done  by 
comparing  dead  reckoning  errors  (e.g.,  difference  between  the  transmitting  player’s 
dead  reckoned  and  interpolated  positions)  to  the  receiving  player’s  entity  state  data 
accuracy  requirements.  If  the  dead  reckoning  errors  are  unacceptable,  there  are  three 
potential  solutions:  (1)  increase  the  order  of  the  dead  reckoning  algorithm  (e.g.,  use 
acceleration  as  well  as  velocity),  (2)  increase  the  generation  rate  of  the  transmitting 
player  to  the  value  required  by  the  receiving  player,  if  possible,  or  (3)  interpolate  the 
transmitting  player’s  entity  state  data  at  the  receiving  node.  (Note  that  the  third 
solution  requires  additional  latency,  since  two  successive  entity  state  data  sets  must  be 
received  before  interpolation  can  be  performed;  hence,  the  receiving  player  is  always 
at  least  one  transmitting  player  update  time  behind;  this  solution  is  only  feasible  if  the 
additional  latency  is  within  required  limits  -  see  latency  requirement  determination 
below.) 

-  Determine  the  time-space-position  information  (TSPI)  accuracy  and  smoothness 
requirements  for  live  players.  This  determination  depends  on  the  test  objectives  and  the 
data  input  requirements  of  the  player  receiving  the  TSPI  data.  For  example,  if  the  TSPI 
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data  will  be  used  as  the  basis  for  evaluating  the  accuracy  of  shooter  targeting  messages, 
the  TSPI  accuracy  should  be  at  least  an  order  of  magnitude  more  accurate  than  the 
targeting  messages.  An  example  of  a  TSPI  smoothness  requirement  is  target  entity  state 
data  being  input  into  a  missile  HWIL  laboratory.  If  the  TSPI  data  input  is  not  sufficiently 
smooth,  the  motion  of  the  missile  flight  table  can  exceed  allowable  limits  and  prevent  the 
missile  from  following  the  target  motion  and  even,  in  extreme  cases,  cause  the  missile 
simulation  to  crash. 

-  Determine  the  requirement  for  data  time-stamp  accuracy.  If  latency  is  to  be  measured,  the 
time  stamps  must  be  accurate  to  the  required  latency  determination  accuracy.  For  most 
applications,  an  accuracy  of  0. 1  milliseconds  is  more  than  adequate. 

-  Also,  the  time  sources  that  determine  the  time  stamps  at  distributed  locations  must  be 
synchronized  to  within  the  required  time-stamp  accuracy.  Techniques  for  achieving 
this  synchronization  requirement  are  discussed  in  References  3  through  5. 
Implementation  of  these  techniques  may  also  impact  hardware  requirements  (e.g., 
Inter-Range  Instrumentation  Group  (IRIG),  global  positioning  system  (GPS))  and 
software  requirements  (e.g.,  network  time  protocol  (NTP)). 

-  During  two  of  the  JADS  tests,  a  time  server  slaved  to  GPS  time  was  used  to 

synchronize  all  loggers.  The  JADS  implementation  used  XNTP,  a  free  software  tool 
from  the  University  of  Delaware  (can  be  downloaded  from 

http://www.eecis.udel.edu/~ntp/).  XNTP  uses  a  UNIX™  daemon,  xntpd,  to 

synchronize  a  local  workstation  clock  to  a  central  time  source.  XNTP  takes  the  local 
clock  performance  into  account  as  it  operates.  The  software  also  considers  the 
network  latencies  and  performance,  thereby  allowing  the  use  of  XNTP  across  LANs 
and  WANs.  During  JADS  testing,  this  technique  allowed  the  time  sources  in  the  data 
loggers,  which  were  separated  by  up  to  a  couple  of  thousand  miles,  to  be 
synchronized  to  within  0.1  milliseconds  of  GPS  time.  More  information  on  XNTP 
can  be  found  in  Reference  5. 

-  Determine  the  classification  of  the  data  and  any  security  handling  requirements  (cost 
impact  factor  of  rank  #6  -  see  Appendix  A).  This  is  generally  driven  by  the  SUT  security 
classification  guide. 

-  Operations  security  (OPSEC)  requirements  for  a  distributed  test  must  be  determined 
and  coordinated  early  in  the  program  especially  given  the  various  organizations 
involved  and  their  different  procedures.  Issues  to  be  addressed  include  the  need  for 
an  OPSEC  plan;  if  an  OPSEC  plan  is  required,  an  agreement  either  to  use  an  already 
approved  OPSEC  plan  or  to  draft  a  new  plan;  the  consistency  of  OPSEC  requirements 
among  the  various  organizations  and  programs;  OPSEC  requirements  in  test 
control/conduct,  including  the  use  of  “For  Official  Use  Only”  test  cards  and/or  the  use 
of  secure  communications. 

-  Document  all  data  exchanges  with  an  interface  control  document  (ICD). 

Data  synchronization  requirements. 

-  Determine  the  requirement  for  synchronizing  multiple  data  inputs  at  a  receiving  node. 
For  example,  shooter  and  target  entity  state  data  may  be  generated  by  different 
simulations  at  different  nodes  and  have  to  be  synchronized  before  input  to  a  missile 
simulation.  As  another  example,  entity  state  data  may  have  to  be  synchronized  to 
messages  at  a  receiving  node. 
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-  Real-time  data  processing  requirements. 

-  Examples  of  data  processing  requirements. 

-  Real-time  data  processing  needed  to  achieve  the  required  TSPI  accuracy  (as 
determined  previously)  for  live  players.  For  example,  during  the  JADS  live  fly  phase 
(LFP)  test  [Ref.  6],  Eglin  Air  Force  Base,  Florida,  developed  a  TSPI  data  processor 
(TDP)  that  merged  several  types  of  TSPI  data  in  near  real  time  (processing  latencies 
of  about  2  seconds).  The  TDP  combined  real-time  aircraft  inertial  navigation  system, 
GPS,  and  ground  tracking  radar  data  to  calculate  more  accurate  kinematics  estimates 
of  the  aircraft. 

-  Real-time  telemetry  processing  limitations.  Certain  live  player  maneuvers  may  screen 
the  telemetry  transmission  path.  Line-of-sight  telemetry  reception  requirements  may 
limit  player  locations/altitudes,  attitude  changes,  and  configuration  (e.g.,  a  live  aircraft 
may  not  be  able  to  carry  external  fuel  tanks  that  could  block  the  telemetry  antenna). 

-  Real-time  data  processing  needed  to  achieve  the  required  synchronization  of  multiple 
data  inputs  at  a  receiving  node.  If  there  are  no  latency  requirements  (such  as  for  the 
case  of  uncoupled,  open-loop  interactions),  buffering  and  time  alignment  of  the  data 
can  be  used.  If  there  are  requirements  to  maintain  low  processing  latencies,  entity 
state  data  can  be  dead  reckoned  to  a  common  data  input  rate. 

-  Real-time  data  processing  needed  to  achieve  latency  compensation.  For  low  latency 
interactions,  entity  state  data  to  be  input  into  a  receiving  simulation  can  be  dead 
reckoned  to  the  current  local  time  in  order  to  compensate  for  latency  effects.  Note 
that  the  clocks  at  the  different  nodes  need  to  be  synchronized  in  order  for  this  latency 
compensation  technique  to  be  effective. 

-  Data  collection/instrumentation  requirements. 

-  Determine  data  collection/instrumentation  requirements  based  on  data  requirements. 

-  Select  instrumentation  types  and  data  logging  software.  Existing  loggers  should  be 
evaluated  carefully  to  ensure  their  adequacy.  JADS  found  that  the  available  loggers 
did  not  have  the  required  time-stamp  accuracy,  and  it  was  necessary  for  JADS  to 
develop  their  own  loggers.  JADS  developed  two  types  of  loggers  for  its  tests. 

-  Passive,  or  stealth,  loggers  were  developed  for  use  at  each  site  during  tests  in 
which  all  data  were  broadcast  among  sites  (e.g.,  for  those  tests  using  distributed 
interactive  simulation  (DIS)  protocol  data  units  (PDUs)). 

-  In-line  loggers  were  developed  for  use  within  each  federate  for  HLA-based 
testing. 

-  Determine  instrumentation  locations. 

-  Determine  the  data  that  must  be  recorded  in  order  to  support  post-test  analysis. 

-  In  some  cases,  the  same  data  may  have  to  be  recorded  at  multiple  locations  in  order  to 
verify  data  integrity  during  transport.  Identify  the  data  source  and  each  required 
recording  location. 

—  Some  data  may  not  be  transported  among  nodes  but  may  still  be  needed  for  analysis 
(e.g.,  internal  simulation  data).  These  data  are  best  recorded  within  the  generating 
player/simulation. 

These  allocated  requirements  are  used  to  assess  the  capabilities  of  the  player  representations 
selected  or  to  determine  design  requirements  for  any  missing  representations.  This  assessment 
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may  reveal  the  need  to  modify  the  selected  representations  (cost  impact  factor  of  rank  #2  -  see 
Appendix  A).  Requirements  for  potential  modifications  are  determined  as  follows. 

-  Determine  if  any  simulation  modifications  will  be  necessary  to  utilize  external  inputs.  These 
may  include  the  following. 

-  Modifications  necessary  to  distribute  incoming  data  to  all  required  hardware  systems  and 
software  programs. 

-  Modifications  necessary  to  input  data  into  simulation  code  in  real  time. 

-  Modifications  necessary  to  meet  synchronization  requirements. 

-  Determine  if  any  simulation  modifications  will  be  necessary  to  generate  required  outputs. 

-  Determine  the  method  to  be  used  for  achieving  the  output  data  time-stamp  accuracy 
requirement.  Also,  it  may  be  necessary  to  time  stamp  internally  logged  simulation  data 
with  absolute  GPS-based  time  stamps  rather  than  the  more  typical  relative  time  stamps  in 
order  to  correlate  data  (GPS  time  stamps  can  provide  a  unique  means  for  tagging  data 
messages)  and  perform  latency  analyses. 

-  Determine  if  any  range  data  processing  modifications  will  be  necessary  to  meet  TSPI 
accuracy,  smoothness,  and  latency  requirements. 

-  Determine  if  any  facility  modifications  will  be  required  for  a  replay  capability  that  can  be 
used  during  integration  testing. 

If  missing  player  representations  must  be  developed,  the  additional  requirements  are  used  for  the 
design  requirements.  The  decision  to  design  and  develop  a  new  representation  can  have  severe 
cost  and  schedule  impacts  on  the  distributed  test,  since  this  can  be  the  single  largest  cost  of  an 
ADS-based  test  (cost  impact  factor  of  rank  #1  -  see  Appendix  A).  For  example,  the  JADS  End- 
to-End  (ETE)  Test  needed  a  ground  emulation  of  the  Joint  STARS  aircraft  radar  subsystem  and 
operator  work  stations,  and  no  adequate  simulation  existed.  Thus,  the  decision  was  made  to 
develop  the  required  simulation,  and  its  cost  was  about  40%  of  the  total  ETE  Test  budget. 

3.3.3  Activity  3.3  -  Prepare  Plan 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  develop  a  coordinated  plan  to 
guide  the  development,  test,  and  execution  of  the  distributed  test.  The  inputs  to  this  activity  are 
the  initial  planning  documents  prepared  during  the  development  of  the  test  objectives  (Activity 
1.2)  and  the  allocated  participant  requirements.  The  output  is  the  detailed  planning  documents. 

The  following  documents  are  needed. 

-  A  detailed  test  plan  that  contains  the  following. 

-  Documentation  of  all  test  objectives  and  requirements. 

-  Descriptions  of  test  events  designed  to  accomplish  the  test  objectives. 

-  Descriptions  of  the  scenarios  to  be  used  during  the  test  events. 

-  The  number  of  trials  to  be  performed  during  each  test  event.  The  number  selected 
should  be  consistent  with  those  needed  to  achieve  desired  confidence  intervals. 

-  Descriptions  of  the  test  configuration  and  instrumentation. 


-  A  plan  for  configuration  management  should  also  be  included.  Configuration 
management  is  a  critical  concern  during  the  design  and  execution  of  a  distributed  test 
and  a  formalized,  disciplined  approach  is  needed  (cost  impact  factor  of  rank  #16  -  see 
Appendix  A).  The  use  of  an  IPT  can  be  an  effective  way  to  maintain  configuration 
control  during  the  planning  phase  of  a  test.  This  should  include  a  methodical 
approach  to  network  management  and  troubleshooting.  Since  problems  are  part  of  the 
process,  the  network  configuration  cannot  be  “frozen”  until  there  is  an  agreed  upon 
“baseline.”  However,  the  configuration  control  process/procedures  must  be 
established  at  the  beginning  of  the  program  and  followed  until  the  end. 

-  Identification  of  any  software  tools  to  be  used  for  configuration  management,  VV&A  (see 
Appendix  B  for  VV&A  activities  to  consider),  or  testing. 

-  Identification  of  resources  needed  to  complete  the  test  including  funding,  personnel, 
equipment  and  materiel,  and  facilities. 

-  Descriptions  of  all  activities  required  to  accomplish  the  test  including  schedules  with  key 
milestones.  Some  considerations  in  setting  up  schedules  include  the  following  (cost 
impact  factor  of  rank  #11  -  see  Appendix  A): 

-  Adequate  time  must  be  allotted  for  data  analysis  between  test  events  during  both 
integration  testing  and  test  execution.  There  is  a  tendency  to  underestimate  the  time 
required  to  adequately  analyze  the  large  volume  of  data  collected  in  distributed  test 
events.  As  a  result,  some  problems  from  one  mission  may  not  be  fully  diagnosed  and 
fixed  before  the  next  mission.  Rehearsal  of  the  analysis  procedures  should  be  used  to 
better  estimate  the  time  required  for  adequate  analysis  between  test  events. 

-  When  scheduling  live  assets,  allowances  must  be  made  for  periodic  military  system 
inspections  (e.g.,  aircraft  phase  inspections),  equipment  breakdown,  and  higher 
priority  missions.  It  is  wise  to  plan  both  a  primary  and  a  contingency  date  for  each 
mission. 

-  Another  consideration  for  live  assets  is  the  use  of  piggyback  missions.  Piggyback 
missions  can  be  used  to  support  integration  testing  and  check-out.  However,  such 
missions  cannot  be  counted  on,  and  backup  dedicated  missions  should  also  be 
scheduled. 

-  The  schedule  should  track  the  WBS.  Scheduling  software  tools,  such  as  Microsoft® 
Project,  are  useful  for  deconflicting  resources  and  for  managing  the  tasks. 

-  Descriptions  of  the  roles  and  responsibilities  of  all  organizations  participating  in  the  test. 

-  A  detailed  DMAP  which  specifies  the  data  requirements,  data  sources,  analysis  procedures, 

and  the  analysis  products  required  to  accomplish  each  test  objective. 

-  Data  reduction  and  analysis  procedures  may  be  aided  by  the  JADS  Analysis  Toolbox.1 

-  A  detailed  VV&A  plan  (see  Appendix  B)  which  includes  integration  testing  to  verify 

architecture  performance  (cost  impact  factor  of  rank  #8  -  see  Appendix  A). 


1  For  further  information  on  the  JADS  Analysis  Toolbox,  please  contact  the  JADS  program  office  or  visit  the  JADS 
website  at  http://www.jads.abq.com  where  it  will  be  available  until  March  2001.  After  that  date,  information  on  the 
toolbox  will  be  available  at  Headquarters  Air  Force  Operational  Test  and  Evaluation  Center  (HQ  AFOTEC)  History 
Office  (HO),  8500  Gibson  Blvd.  SE,  Kirtland  Air  Force  Base,  New  Mexico  87117-5558  and  at  the  Science 
Applications  International  Corporation  (SAIC)  Technical  Library,  2001  North  Beauregard  St.  Suite  800, 
Alexandria,  Virginia  22311. 
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-  A  detailed  ICD  which  specifies  all  data  to  be  passed  between  simulations/range  facilities. 
This  data  specification  includes  data  content,  timing,  formats,  and  coordinate  systems 
(including  all  coordinate  transformation  equations).  The  ICD  will  also  specify  data  bases 
which  must  be  available  at  all  facilities  (see  Activity  4.2).  The  ICD  is  essential  for  ensuring 
successful  integration  of  the  simulations/range  facilities  and  must  be  rigorously  enforced. 

-  If  HLA  is  to  be  implemented,  the  documentation  contained  in  the  Federation  Execution 
Planners  Workbook  (FEPW)  should  be  completed.  See  Reference  1  for  details. 

Detailed  network  design  should  begin  as  soon  as  possible  during  test  planning,  but  no  later  than 
this  activity.  Early  definition  of  network  requirements  allows  for  timely  selection  of  hardware 
and  software  alternatives  and  allows  enough  time  to  acquire  the  right  components  through 
government  channels  and  contracts.  Also,  the  network  requirements  should  be  revisited 
frequently  and  the  design  updated  accordingly  throughout  the  test  planning  process.  Key 
considerations  in  designing  the  network  include  the  following. 

-  Determine  if  data  from  each  node  will  be  broadcast,  multicast,  or  unicast  (transmitted  point- 
to-point).  The  most  common  implementation  of  DIS  uses  broadcasting  of  data.  On  the  other 
hand,  both  DIS  and  HLA  implementation  allow  multicast  or  unicast  only  to  those  nodes 
(federates)  subscribing  to  the  particular  data.  This  determination  impacts  WAN  bandwidth 
requirements,  as  broadcast  requires  the  largest  bandwidth  and  unicast  the  least. 

-  Determine  if  data  are  to  be  transmitted  using  best  effort  or  reliable  procedures  based  on  data 
transport  reliability  requirements.  For  example,  the  user  datagram  protocol  (UDP)  is  not  a 
good  choice  if  highly  reliable  data  transmission  is  needed;  the  transmission  control  protocol 
(TCP)  is  better  for  reliable  transport.  Note  that  reliable  transport  mechanisms  will  result  in 
higher  latency  than  best  effort  transport. 

-  Determine  the  network  security  approach  to  be  implemented  (cost  impact  factor  of  rank  #6  - 
see  Appendix  A). 

-  Designate  a  security  point  of  contact,  perform  a  security  risk  assessment,  and  develop  a 
security  concept  of  operations. 

-  Determine  the  data  encryption  requirements  based  on  the  classification  of  the  data  to  be 
passed  over  the  WAN. 

-  Detailed  procedures  for  secure/encrypted  operations  must  be  developed,  and  approval  for 
their  use  must  be  obtained  from  all  affected  designated  approval  authorities.  Preparations 
for  secure  network  operations  includes  the  following: 

-  Coordinating  the  security  memoranda  of  agreement  (MOAs)  between  the 
organizations  involved.  Allow  3-4  months  for  the  MOA  process. 

-  Obtaining  accreditation  for  networks,  facilities,  rooms,  etc. 

-  Obtaining  a  communications  security  (COMSEC)  account. 

-  Ordering  keying  material  for  encryption  equipment. 

-  Determine  the  WAN  bandwidth  requirement. 

-  Determine  the  average  and  maximum  aggregate  data  rate  by  adding  rates  from  each 
player  transmitted  over  the  WAN.  A  rule  of  thumb  for  the  bandwidth  requirement  is 
given  by  the  aggregate  data  rate  plus  a  50%  -  100%  margin  for  overhead,  bursty  and 
unanticipated  traffic. 
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-  If  the  WAN  is  also  to  be  used  for  voice  communications,  the  anticipated  data  rate  must  be 
added  to  the  aggregate  for  estimating  the  bandwidth  requirement. 

-  Conduct  surveys  of  each  site  to  be  linked  by  the  network. 

-  The  surveys  are  used  to  determine  each  facility’s  communications  architecture  and 
requirements  and  to  determine  the  physical  space  requirements  for  tester-supplied 
equipment  and  personnel.  Two  surveys  are  recommended:  the  first  should  occur  early 
during  Step  3  (design  distributed  test),  and  the  second  should  be  during  this  activity. 

-  The  site  surveys  should  include  the  following. 

-  Establish  points  of  contact  (POCs)  at  each  site. 

-  Determine  the  locations  of  key  network  components,  such  as  the  point  of  presence, 
network  equipment,  etc. 

-  Determine  the  availability  of  space,  power,  and  telephones. 

-  Identify  security  issues,  such  as  POCs,  local  restrictions,  etc. 

3.4  Step  4.  Develop  Distributed  Test 

According  to  the  FEDEP  model,  the  purpose  of  this  step  is  to  develop  the  FOM  (if  HLA  is  to  be 
implemented),  modify  the  simulations/range  facilities  if  necessary,  and  prepare  the  distributed 
architecture  for  integration  and  test.  The  key  activities  for  this  step  and  the  activity  inputs  and 
outputs  are  shown  in  Figure  5  [Ref.  1]. 


Conceptual  Model 


FED  =  federation 


Figure  5.  Develop  Distributed  Test 
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3.4.1  Activity  4.1  -  Develop  FOM 


According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  develop  the  FOM.  The  inputs 
to  this  activity  are  the  detailed  planning  documents  and  the  allocated  participant  requirements. 
The  outputs  are  the  FOM  and  federation  execution  data  (FED)  file,  if  appropriate. 

If  HLA  is  to  be  implemented,  a  FOM  must  be  prepared.  The  FOM  serves  to  document  the  data 
exchanges  required  among  the  federates  to  meet  the  federation  objectives.  Reference  1  outlines 
several  approaches  that  can  be  used  to  develop  the  FOM  (including  use  of  the  HLA  Object 
Model  Data  Dictionary)  and  to  generate  the  FED  file,  and  the  DMSO  HLA  website  contains 
additional  information  and  guidance. 

For  successful  HLA  implementation,  it  is  important  for  key  personnel  to  attend  HLA  training 
classes  early  in  the  program.  This  will  help  clarify  what  HLA  consists  of  and  will  help  unify  the 
vision  of  what  the  HLA  portion  of  the  test  should  entail. 

3.4.2  Activity  4.2  -  Establish  Participant  Agreements 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  establish  all  agreements  among 
participants  necessary  for  a  fully  consistent,  interoperable,  distributed  simulation  environment. 
The  inputs  to  this  activity  are  the  test  scenario,  the  conceptual  model,  and  the  FOM  (if  HLA  is  to 
be  implemented).  The  output  is  revised  participant  allocated  requirements,  including  any 
requirements  for  additional  modifications. 

During  this  activity,  the  participant  interaction  requirements  are  finalized  and  documented  in  the 
ICD  including  the  following. 

-  Determine  the  data  protocols  to  be  used. 

—  Decide  if  standard  protocols  (e.g.,  DIS  PDUs)  are  to  be  used  or  if  it  would  be 
advantageous  to  keep  data  in  formats/coordinate  systems  generated  by  the  players.  If  the 
transmitting  and  receiving  players  use  the  same  coordinate  system  (which  is  different 
from  the  geocentric  DIS  coordinate  system),  processing  time  would  be  increased  by  using 
DIS  PDUs  and  coordinate  transformation  errors  could  result. 

—  Identify  any  data  exchanged  among  players  that  is  best  kept  in  its  tactical  protocol.  For 
example,  if  the  distributed  test  involves  the  T&E  of  an  integrated  launcher/missile 
system,  keeping  the  prelaunch  data  exchanged  between  a  shooter  and  missile  in  its 
tactical  protocol  is  appropriate  for  accomplishing  the  SUT  T&E  objectives  and  eliminates 
the  risk  of  data  corruption  from  unnecessary  protocol  transformations. 

—  Interface  requirements.  Note  that  linking  of  facilities  using  ADS  can  require  significant 

interface  hardware  and  software  development.  ADS  implementation  is  not  “plug  and  play.” 

—  Simulation  interfaces. 

—  Network  interface  units  (NIUs)  of  some  sort  are  necessary  if  two  simulations  cannot 
communicate  directly  in  a  common  language  and  on  a  common  timeline.  The  NIUs 
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interface  the  simulation  output/input  to  the  WAN.  NIUs  need  to  be  carefully 
designed,  since  they  can  be  a  major  source  of  both  error  and  processing  delays. 

-  Determine  coordinate  transformation  requirements  to  convert  from  the  coordinate 
frame  of  the  transmitting  player  to  the  coordinate  frame  of  the  data  protocol  used  over 
WAN  (e.g.,  geocentric  coordinates  for  DIS  PDUs)  and  to  convert  from  the  coordinate 
frame  of  the  data  protocol  used  over  WAN  to  the  coordinate  frame  of  the  receiving 
player. 

-  Determine  dead  reckoning  requirements.  Determine  if  dead  reckoning  will  be  used  at 
the  transmitting  node,  as  well  as  at  the  receiving  node.  Determine  if  second  order  or 
first  order  dead  reckoning  is  required  based  on  position  and  orientation  accuracy 
requirements  and  estimated  latencies  (as  discussed  above). 

-  Determine  requirements  for  simulation  interfaces  to  synchronize  received  data  to 
receiving  players.  Stringent  synchronization  requirements  may  require  that  the 
receiving  simulation  interface  dead  reckon  entity  state  data  to  the  receiving  simulation 
frame  rate. 

-  Special-purpose  interfaces  (cost  impact  factor  of  rank  #3  -  see  Appendix  A). 

-  Determine  if  special  interfaces  will  be  required  to  transfer  synchronized  data  to  SUT 
hardware.  Such  data  may  be  required  to  permit  proper  real-time  operation  of  weapons 
HWIL  labs.  For  example,  the  JADS  LFP  ADS  configuration  required  such  an 
interface  to  link  the  shooter  aircraft-generated  targeting  messages  to  the  HWIL  missile 
in  real  time.  Development  of  such  unique  assets  must  be  anticipated  during  the  test 
concept  development  phase  in  order  to  prevent  schedule  slips. 

-  If  HLA  is  to  be  implemented,  interfaces  will  be  required  between  all  federates  (e.g., 
player  representations,  range  facilities,  etc.)  and  the  runtime  infrastructure  (RTI).  See  the 
DMSO  HLA  website  for  more  details. 

Operational  issues  and  policies  are  also  addressed  and  resolved  including  the  following. 

-  Database  requirements  must  be  determined  and  documented  in  the  ICD. 

-  Determine  requirements  for  a  common  terrain  and  features  database,  including  resolution 
and  level  of  detail. 

-  Determine  requirements  for  any  other  required  common  databases,  such  for  radar  cross 
sections. 

-  Determine  requirements  for  correlating  data  bases  at  distributed  locations. 

-  Post-test  data  management  requirements. 

-  Determine  the  source  and  quantity  of  all  data  to  be  recorded  at  each  recording  location.  A 
distributed  test  with  numerous  trials  can  generate  a  large  volume  of  data  at  distributed 
locations.  Without  careful  planning,  key  data  may  not  be  collected  and/or  transmitted  to 
the  analysis  center,  and  data  collected  at  the  sites  may  not  be  in  a  useful  form  for 
centralized  analysis.  The  DMAP  must  clearly  identify  (1)  the  data  to  be  collected  at  each 
site,  (2)  on-site  processing  of  the  data,  and  (3)  the  data  to  be  transmitted  to  the  analysis 
center  and  data  formats. 

-  Determine  data  handling  procedures  for  collecting  and  storing  data  from  the  distributed 
network.  Determine  if  there  are  any  special  data  handling  requirements,  such  as  for 
classified  or  sensitive  data. 
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-  An  efficient  method  for  retrieving  the  data  to  a  central  analysis  facility  is  by  use  of  the 
network  links.  After  the  test,  data  can  be  retrieved  over  the  network  by  the  use  of  the 
file  transfer  protocol  (FTP)  utility.  This  provides  a  timely,  reliable,  and  secure  data 
retrieval  capability.  (Efficient  retrieval  may  require  that  the  owners  of  the  data  at  each 
site  download  their  files  into  a  central  computer  directory  at  the  site  before  retrieval.) 

-  Use  aggregated  data  quantities  to  determine  the  data  storage  requirements  at  each  site,  as 
well  as  storage  at  the  central  data  analysis  facility.  The  aggregated  quantities  are  used  to 
estimate  the  required  data  storage  capacities.  It  is  recommended  that  additional  capacity 
be  planned  for  (as  a  rule  of  thumb,  plan  for  at  least  twice  the  estimated  data  quantities) 
since  data  quantities  tend  to  grow  as  the  test  progresses. 

-  Using  the  aggregated  data  quantities,  determine  hardware  requirements  for  data  storage 
and  handling  (e.g.,  tape  drives,  compact  disks,  optical  disks). 

Test  control  and  monitoring  requirements  (cost  impact  factor  of  rank  #14  -  see  Appendix  A). 

-  Develop  the  test  control  concept.  This  includes  a  determination  of  the  central  control 
location  and  the  test  coordination  location  at  each  distributed  node/facility.  See 
Appendix  C  for  distributed  test  control  considerations  and  guidelines. 

-  Determine  the  techniques  to  be  used  for  control  of  any  live  players. 

-  For  example,  range  policy  may  dictate  the  use  of  local  air  traffic  controllers  for  live 
aircraft,  so  that  control  of  each  trial  must  be  performed  indirectly  through  them.  (In 
other  words,  the  setup  of  each  trial  would  be  initiated  by  the  distributed  test  director 
who  would  request  that  the  air  traffic  controllers  vector  the  aircraft  appropriately.) 

—  If  remote  live  player  control  is  not  precluded  because  of  range  safety  considerations, 
determine  the  associated  monitoring/communications  requirements.  If  these 
requirements  are  rather  complex  and  costly,  the  best  choice  for  test  control  location 
may  be  at  the  range.  During  the  early  JADS  LFP  risk  reduction  missions  it  became 
apparent  that  when  handling  problems  most  of  the  discussions  centered  on  experts 
and  displays  available  only  at  the  range.  Therefore,  in  order  to  be  part  of  the  decision 
making  process,  the  JADS  test  director  had  to  be  collocated  with  the  Eglin  Air  Force 
Base  range  coordinator  and  the  air  traffic  controllers. 

-  Determine  the  display  and  monitoring  requirements.  This  includes  the  requirements  for 
monitoring  the  status/performance  of  any  live  players,  simulation  facilities,  and  the 
distributed  network.  Two-dimensional  displays  of  the  common  synthetic  battlespace  are 
recommended  at  each  node.  This  will  greatly  improve  the  situational  awareness  of  the 
participants  at  each  site.  During  JADS  System  Integration  Test  (SIT),  the  displays 
converted  entity  state  data  into  symbols  which  were  overlaid  on  a  local  area  map.  As  a 
result,  the  test  team  members  knew  where  and  in  what  direction  the  aircraft  were  flying 
and  if  the  missile  simulation  was  active. 

-  Determine  the  voice  communications  requirements  for  effective  control  and  monitoring 
of  the  distributed  test.  Determine  how  many  separate  voice  networks  will  be  needed  and 
how  many  individuals  will  be  plugged  into  each  network.  Use  these  requirements  to 
evaluate  the  adequacy  of  existing  telephone  systems  to  provide  “loud  and  clear” 
communications.  If  existing  telephone  systems  are  judged  to  be  inadequate,  an  option  is 
to  reserve  a  portion  of  the  WAN  bandwidth  for  voice  communications. 

—  Separate  voice  networks  are  advisable  whenever  certain  groups  in  the  test  force  only 
need  to  communicate  among  themselves.  Examples  are  communications  between  the 
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player  operators  and  the  player  controllers,  communications  between  the  test 
controller  and  facility/instrumentation  operators,  and  communications  among  the 
decision  makers  at  the  sites.  Limiting  the  personnel  that  can  actively  transmit  on  each 
network  (using  push-to-talk  switches)  results  in  clearer  communications  (others  can 
listen  on  speakers). 

-  Existing  telephone  systems  operated  in  a  conference  call  mode  usually  can  only 
accommodate  a  limited  number  of  people  with  voice  qualities  that  can  vary 
significantly.  Also,  during  conference  calls  over  Defense  Switched  Network  (DSN) 
phone  lines,  callers  can  be  randomly  disconnected. 

3.4.3  Activity  4.3  -  Implement  Participant  Modifications 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  implement  participant 
modifications  identified  in  previous  activities.  The  input  to  this  activity  is  the  updated  allocated 
participant  requirements.  The  output  is  the  modified  participants. 

The  ADS  test  network  should  also  be  installed  during  this  activity  using  the  following  steps  (cost 
impact  factor  of  rank  #5  -  see  Appendix  A). 

-  The  WAN  to  be  used  to  link  the  facilities  is  selected  and  procured. 

-  This  selection  is  based  on  a  determination  of  whether  a  Department  of  Defense  (DoD)- 
sponsored  network  can  support  the  test  requirements  or  if  commercial  leased  lines  must 
be  used. 

-  First  the  network  requirements  previously  developed  are  consolidated,  including 
bandwidth  requirements,  acceptable  latency  limits,  aggregate  data  rates, 
availability/reliability  requirements,  and  network  management/control  requirements. 

-  Next  the  requirements  are  submitted  to  Headquarters,  Defense  Information  Systems 
Agency,  Defense  Information  Systems  Network  (DISN)  Program  Office  for  a 
determination  of  whether  DISN  common-user  services  (e.g..  Defense  Simulations 
Internet  (DSI),  Secret  Internet  Protocol  Router  Network  (SIPRNET))  will  support  the 
requirements  or  if  a  waiver  to  policy  is  justified  [Ref.  7].  Note  that  there  is  a 
minimum  lead  time  of  180  days  for  acquiring  common-user  networks. 

-  If  a  waiver  to  policy  is  justified,  then  commercial  line  lease  rates  are  surveyed  to 
determine  if  dedicated  (full-time)  leased  lines  are  required  or  if  on-demand  leased  lines 
will  suffice  (cost  impact  factor  of  rank  #13  -  see  Appendix  A).  DoD  agencies  are 
required  to  contract  for  leased  lines  through  the  DISN  Program  Office. 

-  This  determination  is  usually  based  on  test  schedule  and  cost  considerations.  If  at 
least  weekly  use  of  the  network  is  anticipated  during  the  ADS  architecture  integration 
and  test  and  test  execution  phases,  then  it  is  generally  more  cost  effective  to  contract 
for  dedicated  lines. 

-  Once  a  decision  has  been  made  on  dedicated  versus  on-demand  lines,  a  contract  for 
the  leased  lines  must  be  processed.  The  contract  processing  time  can  be  rather 
lengthy  (up  to  six  months),  so  that  adequate  lead  time  is  needed  to  maintain  the  test 
schedule.  Note  that  contracts  for  commercial  leased  lines  can  be  somewhat  expensive 
and  can  have  difficult  termination  clauses.  For  example,  the  contract  with  American 
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Telephone  and  Telegraph  (AT&T)  to  provide  the  T-l  line  for  the  JADS  LFP  test  cost 
$4448  per  month  with  a  minimum  12-month  lease,  required  a  60-day  notice  to 
terminate  the  line,  and  had  a  $10,000  penalty  for  early  termination  of  the  contract. 

The  network  hardware  to  be  used  is  selected,  procured,  and  installed,  including  routers, 
channel  service  units  (CSUs)/data  service  units  (DSUs),  multiplexes,  encryption  equipment, 
etc.  (cost  impact  factor  of  rank  #13  -  see  Appendix  A). 

-  It  is  advisable  to  use  standard  network  equipment  and  to  select  the  same  brand  of  router 
hardware  and  software  for  all  network  nodes.  JADS  experience  is  that  the  lack  of 
common  router  hardware  and  software  can  result  in  numerous  interoperability  problems. 
During  the  JADS  linked  simulators  phase  (LSP)  test  [Ref.  8],  it  was  difficult  to  get  the 
network  to  behave  in  a  uniform  fashion  because  of  the  many  different  types  of  interface 
hardware,  routers,  and  interface  software  versions.  Also,  lack  of  common  software 
resulted  in  the  NIUs  having  numerous  problems  related  to  conversions,  timing,  central 
processor  unit  (CPU)  speed,  etc. 

-  The  router  addressing  scheme  also  needs  to  be  determined  (see  Ref.  3  for  details). 

-  Acquisition  of  encryption  equipment  and  keying  materials  requires  a  minimum  of  90  days 
if  the  account  is  already  established. 

-  The  use  of  a  network  test  bed  is  recommended  prior  to  installing  and  checking  out  the 
actual  network.  This  allows  an  early  test  of  the  network  design  and  its  implementation, 
along  with  a  check-out  of  the  networking  equipment.  Using  this  approach,  installation  of 
the  actual  network  becomes  almost  “plug  and  play,”  and  the  operators  get  early  training 
and  familiarization. 

The  interfaces  necessary  for  linking  are  built/procured  in  accordance  with  the  requirements 
developed  during  Activity  4.2. 

Test  control  hardware  and  software  are  selected,  procured,  and  installed  based  on  the 
requirements  developed  in  Activity  3.1  (cost  impact  factor  of  rank  #14  -  see  Appendix  A). 

-  The  selection  criteria  include  both  data  and  voice  communications  requirements. 

-  Test  control  software  includes  that  needed  for  player  visualization  at  the  test  control 
center.  If  entity  state  data  are  to  be  transmitted  to  the  center  in  the  form  of  DIS  PDUs, 
commercial  and  government  off-the-shelf  software  are  available  for  this  purpose. 

-  The  test  control  center  physical  space  requirements  and  layout  are  also  determined. 
Network  analysis/monitoring  tools  are  selected,  procured  and/or  developed,  and  installed. 
Considerations  here  are  as  follows  (cost  impact  factor  of  rank  #15  -  see  Appendix  A). 

-  The  selection  of  tools  depends  on  the  desired  extent  of  the  analysis/monitoring.  For 
example,  diagnostic/troubleshooting  tools  are  different  from  those  for  analyzing  network 
performance. 

-  Common  practice  is  to  monitor  bandwidth  utilization.  In  this  case,  tool  selection  is 
driven  by  such  considerations  as  whether  all  links  are  to  be  monitored  versus  only  some 
of  the  links  and  whether  local  area  networks  (LANs)  will  be  monitored  locally  or  from  a 
central  remote  location.  JADS  used  two  network  tools:  Cabletron  Systems,  Inc., 
SPECTRUM®  to  monitor  bandwidth  utilization  (more  information  at 
http://www.cabletron.com/spectrum/)  and  Silicon  Graphics,  Inc.,  NetVisualyzer™  to 
monitor  packet  rate  and  packet  contents  (more  information  at  http://www.sgi.com). 
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-  The  JADS  Electronic  Warfare  (EW)  Test  also  used  link  health  messages  to  monitor 
federate  and  communication  link  status.  Each  federate  would  broadcast  its  own  link 
health  status  to  all  other  federates  at  a  rate  of  once  per  second.  Details,  including  the 
message  formats,  are  given  in  Reference  9. 

-  Before  monitoring  or  collecting  data  from  a  LAN  under  the  control  of  another 
organization,  it  will  be  necessary  to  obtain  permission  from  the  appropriate  authority. 

3.5  Step  5.  Integrate  and  Test  Architecture 

According  to  the  FEDEP  model,  the  purpose  of  this  step  is  to  plan  the  test  execution,  establish  all 

required  interconnectivity  between  the  nodes/players,  and  test  the  network  prior  to  execution. 

The  key  activities  for  this  step  and  the  activity  inputs  and  outputs  are  shown  in  Figure  6  [Ref.  1], 


Modified  Participants 


Figure  6.  Integrate  and  Test  Architecture 
3.5.1  Activity  5.1  -  Plan  Execution 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  define  and  develop  the  full  set 
of  information  required  to  support  the  distributed  test  execution.  The  inputs  to  this  activity  are 
the  FOM  and  the  FED  file,  if  appropriate,  the  test  scenario,  and  the  detailed  planning  documents. 
The  outputs  are  a  refined  and  detailed  integration  test  plan  (cost  impact  factor  of  rank  #8  -  see 
Appendix  A),  VV&A  plan  (see  Appendix  B),  and  test  procedures. 

The  test  and  VV&A  plans  (see  Appendix  B  for  VV&A  considerations)  should  be  refined, 
especially  the  integration  test  plan  section  (cost  impact  factor  of  rank  #8  -  see  Appendix  A).  In 
refining  the  integration  test  plan,  step-by-step  systematic  integration  testing  procedures  need  to 
be  developed  and  should  address  the  following. 

-  Procedures  for  verifying  any  simulation/range  facility  modifications  or  any  new  player 
representations  in  a  systematic  stand-alone  fashion. 

-  Procedures  for  initially  testing  each  WAN  link  separately.  Testing  should  begin  at  the  CSU/ 
DSU  level  to  make  sure  that  communications  work  at  the  lowest  level. 

-  The  complete  simulation-to-simulation  (or  live  player-to-simulation)  connection  should 
be  tested  for  each  link. 
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-  Coordinate  transformations  must  be  verified  at  each  site  and  then  reverified  during  end- 
to-end  testing.  Personnel  who  are  subject  matter  experts  in  coordinate  transformations 
must  be  assigned  and  readily  available  during  this  process. 

-  The  actual  data  protocols  to  be  used  during  test  execution  should  be  used  to  verify 
message  routing  and  check  network  loading. 

-  Structured  testing  of  the  network  must  be  performed  prior  to,  and  independent  of,  the 
linked  testing  times  in  order  to  verify  transmission/reception  rates,  bandwidth  utilization, 
latency,  etc.  “Pings”  are  useful  to  check  for  connectivity  and  loading  problems. 

-  Procedures  for  testing  each  simulation-to-simulation  connection  with  all  network  nodes 
connected.  Network  analysis/monitoring  should  be  used  to  troubleshoot  the  network. 
Special  test  equipment  (e.g.,  data  loggers)  and  networking  tools  (e.g.,  SPECTRUM®, 
EtherPeek™)  are  useful  for  more  rapidly  isolating  and  determining  the  cause  of  network 
problems.  Without  these,  trial  and  error  becomes  the  normal  troubleshooting  mode  which 
increases  the  resource  requirements  (time,  schedule,  cost,  etc.). 

-  Procedures  for  testing  the  voice  communications  with  all  equipment  and  personnel  as  in  the 
actual  test. 

Also,  detailed  test  control  procedures  and  a  security  test  and  evaluation  plan  should  be 
developed.  The  test  control  procedures  should  include  checklists  covering  activities  for  24  hours 
prior,  4  hours  prior,  1  hour  prior,  during  the  mission,  and  post-test.  Examples  can  be  found  as 
appendices  to  the  JADS  test  reports.2  Existing  stand-alone  facility/range  procedures  serve  as  a 
starting  point  for  developing  the  checklists.  The  system  integrator  uses  these  to  develop  new 
checklists  which  interleave  the  activities  at  each  facility  and  include  those  unique  to  the 
distributed  test  environment. 

3.5.2  Activity  5.2  -  Integrate  and  Test  ADS  Architecture 

This  activity  combines  the  separate  FEDEP  activities  of  “integrate  federation”  and  “test 
federation”  because  of  the  close  connection  between  the  two.  An  iterative  “test-fix-test” 
approach  is  recommended,  so  that  the  integration  and  test  activities  become  closely  interrelated. 
According  to  the  FEDEP  model,  the  purpose  of  these  activities  is  to  bring  all  the  distributed  test 
participants  into  a  unifying  operating  environment  and  to  test  that  they  can  all  interoperate  to  the 
degree  required  to  achieve  the  test  objectives.  The  inputs  to  this  activity  are  the  detailed 
integration  test  plan,  VV&A  plan,  and  test  procedures.  The  outputs  are  refined  test  procedures 
(to  be  used  during  test  execution),  W&A  results,  and  an  ADS  architecture  that  has  been 
thoroughly  tested  and  is  ready  for  test  execution. 

Key  testing  steps  during  this  activity  include  the  following. 


2  Reports  for  each  phase  of  JADS  testing  may  be  found  at  the  JADS  website  at  http://www.jads.abq.com  until  March 
2001.  After  that  date,  JADS  reports  will  be  available  at  HQ  AFOTEC/HO,  8500  Gibson  Blvd  SE,  Kirtland  Air 
Force  Base,  New  Mexico  87117-5558  and  at  the  SAIC  Technical  Library,  2001  North  Beauregard  St.  Suite  800, 
Alexandria,  Virginia  22311. 
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Perform  compliance  testing,  as  specified  in  the  VV&A  plan. 

-  Test  each  facility/node  individually  to  ensure  that  ADS  capability  and  any  required 
modifications  (including  software)  have  been  correctly  implemented.  Specialized  tools 
for  reading  and  analyzing  raw  simulation/facility  data  are  needed  for  this  verification. 

-  Acceptance  testing  of  any  software  developed  for  the  test  is  included  in  this  step. 

Perform  integration  testing,  as  specified  in  the  integration  test  plan. 

-  Check  out  interfaces  and  facility  modifications  with  linking  between  pairs  of  nodes. 

-  Baseline  the  performance  of  the  network  with  no  loading  from  the  simulations/players. 

-  As  discussed  above,  the  use  of  a  network  test  bed  is  recommended  prior  to  installing 
and  checking  out  the  actual  network. 

-  If  the  use  of  a  test  bed  is  not  possible,  then  the  networking  equipment  should  be  tested 
individually  first  and  should  be  preconfigured  whenever  possible. 

-  Test  performance  of  critical  portions  of  the  network  under  loading  representative  of  test 
conditions  to  be  used.  Voice  communications,  as  well  as  data  processing  and 
transmission,  should  be  included  in  the  loading  conditions.  This  period  should  also  be 
used  to  experiment  with  reliable  versus  best  effort  data  transport  modes  in  order  to 
determine  the  optimum  mix  which  balances  lower  latencies  (using  best  effort)  with  lower 
data  losses  (using  reliable).  Useful  testing  tools  for  this  process  include  the  following: 

-  Communications  testers  for  bit  error  rate  testing  (e.g.,  Fireberd  testers). 

-  Protocol  sniffers. 

-  Simple  Network  Management  Protocol  (SNMP)  tools  are  not  critical  but  are  nice  to 
have. 

-  Custom  tools,  such  as  displays  driven  by  link  health  check  messages. 

-  Use  an  iterative  “test-fix-test”  approach,  including  replay  of  trials  to  diagnose  problems 
and  verify  fixes.  Trial  replays  are  efficient  and  cost  effective  during  this  process  and 
involve  replaying  input  data  recorded  from  the  fully  linked  tests  into  critical  interfaces. 
However,  not  all  stand-alone  facilities  are  designed  to  accept  replay  inputs  and  some 
modifications  may  be  required.  It  may  also  be  necessary  to  develop  a  special  data  player 
for  this  purpose. 

-  Replays  are  also  an  excellent  method  for  rehearsing  and  refining  test  procedures  and 
for  working  out  technical  and  procedural  issues. 

Perform  risk  reduction  missions.  The  purpose  of  these  fully  linked  missions  is  to  exercise  all 
parts  of  the  distributed  test  to  ensure  that  they  operate  as  intended.  The  early  risk  reduction 
missions  are  invaluable  for  identifying  problems  before  the  actual  test.  The  later  missions  are 
used  to  verify  fixes  and  serve  as  rehearsals  for  the  formal  test  execution. 

-  Execute  the  scenarios  with  the  fully  linked  test  execution  configuration  (using  live 
players,  if  appropriate)  and  with  all  locations  manned  as  in  the  actual  test. 

-  Include  security  certification,  if  required. 

-  Evaluate  test  control  and  monitoring  procedures,  and  modify  the  procedures,  as 
appropriate,  for  the  next  risk  reduction  mission. 

-  Execute  data  collection  and  analysis  procedures,  and  modify  the  procedures,  as 
appropriate,  for  the  next  risk  reduction  mission.  Getting  the  data  during  the  risk  reduction 
missions  is  beneficial  for  developing  and  verifying  analysis  tools  before  the  formal  test 
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execution  starts.  During  the  JADS  LFP  test,  this  early  exposure  to  the  data  gave  the 
programmers  and  analysts  time  to  read  the  raw  data  formats,  choose  among  statistical 
packages,  build  real-time  data  display  routines,  make  utilities  to  convert  units,  and 
rehearse  analysis  procedures. 

-  Perform  validation,  as  specified  in  the  VV&A  plan. 

3.6  Step  6.  Execute  Distributed  Test  and  Analyze  Results 

According  to  the  FEDEP  model,  the  purpose  of  this  step  is  to  execute  the  distributed  test,  process 
the  output  data  from  the  test  execution,  report  results,  and  archive  reusable  test  products.  The 
key  activities  for  this  step  and  the  activity  inputs  and  outputs  are  shown  in  Figure  7  [Ref.  1]. 


Figure  7.  Execute  Distributed  Test  and  Analyze  Results 
3.6.1  Activity  6.1  -  Execute  Distributed  Test 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  exercise  all  distributed  test 
participants  as  an  integrated  whole  to  generate  required  outputs  and  thus  achieve  the  stated  test 
objectives.  The  inputs  for  this  activity  are  the  refined  test  procedures  and  tested  ADS 
architecture  from  integration  testing.  The  output  is  the  raw  test  results. 

The  main  function  of  this  activity  is  to  execute  the  test  matrix  detailed  in  the  test  plan.  During 
execution,  the  following  considerations  apply: 

-  Pre-  and  post-test  briefings  are  essential. 

-  A  premission  briefing  should  be  held  24  hours  before  each  mission  and  is  critical  for 
coordinating  the  many  network  and  test  issues.  The  briefing  agenda  should  include  the 
test  objectives,  planned  test  matrix,  personnel  involved,  telephone  numbers/frequencies  to 
use  for  test  control,  go/no  go  criteria,  contingency  plans  in  case  of  failures, 
instrumentation  and  data  collection  requirements,  details  on  facility  configurations, 
OPSEC,  and  the  time  and  place  of  the  debrief.  Tactical  equipment  operators  (e.g.,  air 
crews)  should  be  involved  and  thoroughly  briefed  on  the  scenarios/profiles  and  their 
checklist  items.  If  possible,  the  same  operators  should  be  used  for  each  mission. 

—  Deploy  test  force  personnel  to  the  range/sites  several  hours  prior  to  the  mission  to  confirm 
that  the  preparations  have  been  completed. 
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-  Centralized  test  control/management  and  execution  monitoring  must  be  maintained, 
following  detailed  test  control  procedures  and  checklists.  Go/no  go  criteria  and  contingency 
plans  must  be  developed  in  advance. 

-  Missions  with  live  players  require  more  contingency  planning  in  order  to  quickly  decide 
on  alternatives.  Good  contingency  planning  will  allow  the  test  director  to  make  rapid, 
well-informed  decisions  and  get  the  most  productive  use  out  of  remaining  range  time. 

-  Detailed  test  cards  should  be  used  for  each  mission.  The  test  cards  should  specify  the 
profile(s)  to  be  used  for  each  trial  and  the  configuration  of  each  player.  Back-up  test 
cards  should  also  be  prepared  for  contingency  purposes. 

-  Data  are  collected  during  execution  to  support  both  quick-look  and  detailed  post-test 
analyses.  The  quick-look  data  are  used  by  the  analysts  and  SUT  experts  after  each  trial  to 
ensure  that  the  test  objectives  are  being  accomplished.  Time  must  be  allocated  at  the  end  of 
each  test  period  for  data  logging,  data  archiving,  data  transfer,  and  facility  reclassification  (a 
minimum  of  two  additional  hours  was  needed  during  the  JADS  SIT). 

-  Strict  attention  must  be  given  to  maintaining  the  security  posture  of  the  ADS  architecture 
during  execution. 

-  Conduct  an  after-action  review  immediately  after  the  test  in  order  to  gather  important 
information  from  each  facility,  to  formulate  a  course  of  action  for  correcting  any  problems, 
and  to  prepare  for  the  next  period  of  testing. 

3.6.2  Activity  6.2  -  Process  Output 

According  to  the  FEDEP  model,  the  purpose  of  this  activity  is  to  post-process  (as  necessary)  the 
output  collected  during  the  test  execution.  The  input  to  this  activity  is  the  raw  test  results  from 
test  execution.  The  output  is  derived  test  results. 

The  data  are  analyzed  in  accordance  with  the  DMAP.  The  analysis  should  include  the  network 
performance,  the  SUT  performance,  and  the  impact  of  the  network  on  the  SUT  performance. 
Statistical  measures  are  applied,  if  appropriate,  and  other  data  reduction  methods  are  used  to 
transform  the  raw  data  into  derived  results.  Error  estimates  due  to  inaccuracies  in  measurements 
and  sampling  are  determined. 

3.6.3  Activity  6.3  -  Prepare  Results 

According  to  the  FEDEP  model,  this  activity  has  two  purposes:  (1)  to  evaluate  the  data  analysis 
results  in  order  to  determine  if  all  test  objectives  have  been  met,  and  (2)  to  identify  legacy 
products  and  make  them  available  to  other  programs.  The  input  to  this  activity  is  the  derived  test 
results,  along  with  the  test  evaluation  criteria  from  Activity  2.3.  The  outputs  are  documented  test 
results  and  legacy  products. 

Examples  of  legacy  products  include  lessons  learned,  the  final  distributed  architecture 
configuration,  the  distributed  architecture  performance  analysis,  and  recommendations  for  future 
distributed  test  implementations.  Detailed  documentation  of  the  distributed  architecture 
configuration  is  important  for  potential  replication  of  the  configuration  in  future  tests. 
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3.6.4  Return  to  Test  Concept  Development  Process 

If  there  are  additional  operational  tasks  relevant  to  the  SUT,  then  the  test  planners  need  to  return 
to  Step  10  of  the  concept  development  process  in  Section  2.0  of  this  document.  If  there  are  no 
additional  tasks,  then  the  planning  and  execution  process  is  complete. 
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4.0  Conclusion 


This  special  report  outlines  the  steps  in  planning  and  implementing  ADS-based  testing.  The  test 
concept  development  methodology  guides  the  tester  in  making  the  decision  on  whether  to 
implement  ADS-based  testing;  the  test  planning  and  implementation  methodology  uses  JADS 
lessons  learned  to  illustrate  the  activities  embodied  in  the  FEDEP  model. 

Although  the  methodology  steps  are  presented  in  a  sequential  fashion,  experience  has  shown  that 
many  of  the  activities  are  actually  cyclic  with  extensive  feedback  between  activities  and/or 
concurrent.  Implementers  should  not  enforce  a  strict  waterfall  approach  to  the  steps  given.  Not 
only  may  variations  in  the  order  of  the  activities  be  appropriate,  but  it  is  frequently  necessary  to 
revisit  previous  activities  as  the  distributed  test  requirements  and  design  become  more  mature. 
The  methodology  given  here  must  be  tailored  to  each  specific  distributed  test  implementation. 
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6.0  Acronyms  and  Definitions 


AAA 

anti-aircraft  artillery 

ABCCC 

airborne  battlefield  command  and  control  center 

ABL 

airborne  laser 

ADS 

advanced  distributed  simulation 

ANGLICO 

air  and  naval  gunfire  liaison  company 

AOC 

air  operations  center 

AT&T 

American  Telephone  and  Telegraph 

AWACS 

Airborne  Warning  and  Control  System 

CAS 

close  air  support 

CEA 

circular  error  average 

CEP 

circular  error  probable 

COI 

critical  operational  issue 

comjam 

communications  jamming 

COMSEC 

communications  security 

CONOPS 

concept  of  operations 

COTS 

commercial-off-the-shelf 

CPU 

central  processor  unit 

csu 

channel  service  unit 

CTOC 

corps  tactical  operations  center 

DIS 

distributed  interactive  simulation 

DISN 

Defense  Information  Systems  Network 

DMAP 

data  management  and  analysis  plan 

DMSO 

Defense  Modeling  and  Simulation  Office 

DoD 

Department  of  Defense 

DSI 

Defense  Simulations  Internet 

DSM 

digital  system  model 

DSN 

Defense  Switched  Network 

DSU 

data  service  unit 

DT&E 

developmental  test  and  evaluation 

ETE 

JADS  End-to-End  Test 

EW 

electronic  warfare;  JADS  Electronic  Warfare  Test 

FAC 

forward  air  controller 

FAC-A 

forward  air  controller-air 

FED 

federation  execution  data 

FEDEP 

federation  development  and  execution  process 

FEPW 

federation  execution  planners  workbook 

FO 

forward  observer 

FOM 

federation  object  model 

FTP 

file  transfer  protocol 

GPS 

global  positioning  system 

HLA 

high  level  architecture 

HQ  AFOTEC 

Headquarters,  Air  Force  Operational  Test  and  Evaluation  Center 
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HQ  DISA 

Headquarters,  Defense  Information  Systems  Agency 

HWIL 

hardware-in-the-loop 

ICD 

interface  control  document 

IPT 

integrated  product  team 

IRIG 

Inter-Range  Instrumentation  Group 

JADS 

Joint  Advanced  Distributed  Simulation 

Janus 

interactive,  computer-based  simulation  of  combat  operations 

Joint  STARS 

Joint  Surveillance  Target  Attack  Radar  System 

LAN 

local  area  network 

LFP 

live  fly  phase 

LSP 

linked  simulators  phase 

MITL 

man-in-the-loop 

MOA 

memorandum  of  agreement 

MOE 

measure  of  effectiveness 

MOP 

measure  of  performance 

NIU 

network  interface  unit 

NTP 

network  time  protocol 

OML 

Object  Model  Library 

OPSEC 

operations  security 

ORD 

operational  requirements  document 

OT&E 

operational  test  and  evaluation 

PDU 

protocol  data  unit 

POC 

point  of  contact 

RCM 

requirements  correlation  matrix 

ROM 

rough  order  of  magnitude 

RTI 

runtime  infrastructure 

SAIC 

Science  Applications  International  Corporation 

SAM 

surface-to-air  missile 

SIPRNET 

Secret  Internet  Protocol  Router  Network 

SIT 

JADS  System  Integration  Test 

SME 

subject  matter  expert 

SNMP 

Simple  Network  Management  Protocol 

SOC 

statement  of  capability 

SOM 

simulation  object  model 

SPECTRUM® 

a  network  analysis  package  developed  by  Cabletron  Systems 

STAR 

system  threat  assessment  report 

SUT 

system  under  test 

T&E 

test  and  evaluation 

TACCSF 

Theater  Air  Command  and  Control  Simulation  Facility,  Kirtland  AFB,  New 
Mexico 

TAOC 

tactical  air  operations  center 

TCP 

transmission  control  protocol 

TDP 

TSPI  data  processor 

TOC 

tactical  operations  center 

TSPI 

time-space-position  information 
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UDP 

user  datagram  protocol 

VTC 

video  teleconference 

v&v 

verification  and  validation 

VV&A 

verification,  validation,  and  accreditation 

WAN 

wide  area  network 

WBS 

work  breakdown  structure 

woe 

wing  operations  center 
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Appendix  A  -  Cost  Factors  for  ADS-Based  Testing 


This  table  is  intended  to  aid  an  estimator  in  creating  a  detailed  cost  estimate  for  an  advanced 
distributed  simulation  (ADS)-based  test  and  focuses  on  the  unique  aspects  of  an  ADS-based  test. 


Impact 

Rank 

Cost  Factor 

Risk  to  Estimate/Expected 
Impact  to  Total  Cost 

Recommended  Estimator 
Response 

1 

Design  and  develop 
missing  player 
representation* 

High/High  -  Player 
representations  can  vary  in  fidelity 
and  complexity  from  simple  script- 
driven  representations  to  high 
fidelity,  man-in-the-loop 
simulators  or  live  assets. 
Development  of  a  new  player 
representation  can  be  the  largest 
cost  to  an  ADS  test. 

Request  a  bottom-up  estimate  from 
those  most  knowledgeable  for  each 
missing  player.  Special  risk 
factors  to  estimate  are  stability  of 
requirements,  special 
instrumentation  requirements,  and 
verification,  validation  and 
accreditation  (VV&A) 
requirements.  Software 
models/script-driven 
representations  may  be  estimated 
using  standard  parametric  software 
models  once  a  size  estimate  is 
made.  Special  risk  factors  to 
estimate  are  W&A  requirements 
and  maturity  of  developer’s 
software  development  process. 

2 

Modify  existing  player 
representation* 

High/  High  -  Modifications  can 
range  from  simple  to  extensive. 

Request  detailed  estimate  from 
simulation/range  facility 
developer.  Expect  10-15%  over 
run  fr  om  government 
organizations  (based  on  the 

System  Integration  Test  (SIT) 
experience).  Special  risk  factors 
to  estimate  are  stability  of 
requirements,  special 
instrumentation  requirements, 
VV&A  requirements,  and  maturity 
of  developer’s  software 
development  process. 

3 

Design  and  develop 

special-purpose 

interfaces 

High/High  -  Special-purpose 
interfaces  indicate  high 
performance  in  some  dimension  is 
required  and  is  critical  to 
distributed  test  success. 

Request  bottom-up  estimate  from 
most  knowledgeable  source. 

Special  risk  factors  to  estimate  are 
stability  of  requirements,  special 
instrumentation  requirements, 
VV&A  requirements,  maturity  of 
hardware/protocol  technology. 

^Examples  of  player  representations  in  an  ADS-based  test  are  digital  simulations,  hardware-in- 
the-loop  (HWIL)  labs,  range  facilities,  and  instrumented  live  assets. 


45 


Impact  Cost  Factor 
Rank  _ 


Network  architecture 
complexity 


System  engineering 
process/strong  system 
integrator 


Risk  to  Estimate/Expected 
Impact  to  Total  Cost 


Medium/  Medium  -  Low 
transmission  latency  (<100 
milliseconds)  requires  additional 
testing  and  development  time  to 
ensure  architecture  is  tuned.  Also 
increases  ADS  architecture 
integration  and  testing  time.  May 
require  more  expensive  computers 
(multiprocessors)  and  network 
hardware.  Increases  the  risk  that 
simulation/range  facility  modification 
may  be  necessary.  Increases  the 
likelihood  that  ADS  network 
instrumentation  development  is 
necessary. 


Medium/  Medium  -  Complexity  is 
measurable  in  number  of  nodes, 
number  of  links,  expected  bandwidth 
used.  High  complexity  in  any  of 
these  requires  additional  architecture 
testing  and  development  time  to 
ensure  architecture  is  tuned.  Also 
increases  integration  and  testing  time. 
May  require  more  expensive 
computers  (multiprocessors)  and 
network  hardware.  Increases  the  risk 
that  player  representation 
modification  may  be  necessary. 
Increases  the  likelihood  that  ADS 
network  instrumentation  development 
is  necessary. 


Medium/  Low  -  Primary  risk  is 
connecting  at  higher  levels  of 
security  than  facility/simulation 
normally  operates.  May  require 
modifications  to  affected  facilities. 
May  increase  operating  costs  of 
facilities.  Other  risks  are  in 
implementing  multiple  security  levels 
in  the  same  facility.  This  may  require 
special  hardware  or  procedures. 


Medium/  Medium  -  Poor  history  or 
weak  systems  engineering  process 
will  increase  overall  cost  and 
schedule  risk.  Weak  systems 
integration  will  increase  design, 
development,  and  integration  time. 
Also  increases  requirements 
volatility. 


Recommended  Estimator 
Response 


Increase  risk  to  integration 
schedule  and  budget  for  hardware 
upgrades.  Investigate  need  for 
simulation/range  facility 
modification  and  developing  ADS 
network  instrumentation.  Unstable 
requirements  add  to  the  risks 
imposed  by  this  factor. 


Increase  risk  to  integration 
schedule  and  budget  for  hardware 
upgrades.  Investigate  need  for 
simulation/range  facility 
modification  and  developing  ADS 
network  instrumentation.  Unstable 
requirements  add  to  the  risks 
imposed  by  this  factor. 


Increase  risk  to  integration 
schedule  and  budget  for  hardware 
upgrades.  Investigate  need  for 
federate  modification  and 
developing  ADS  instrumentation. 
Unstable  requirements  add  to  the 
risks  imposed  by  this  factor. 


Increase  risk  to  overall  schedule 
and  budget.  Presence  of  poor 
systems  engineering  practices  or 
weak  systems  integration  will 
increase  elements  affected  by 
requirements  stability. 
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Impact 

Rank 

Cost  Factor 

Risk  to  Estimate/Expected 
Impact  to  Total  Cost 

Recommended  Estimator 
Response 

8 

Integration  and  testing 

Medium/Medium  -  Integration  and 
testing  can  be  expected  to  reveal 
unforeseen  problems  which  must  be 
resolved  before  the  distributed 
architecture  is  ready  for  the  execution 
phase. 

Do  not  rely  on  success-oriented 
integration  schedule  and  budget. 

Plan  on  additional  time  and  money 
for  troubleshooting.  Add 
additional  risk  reduction  missions 
for  testing  anticipated  fixes. 

9 

Maturity  of 
requirements 

Medium/  Medium  -  This  is  a 
reflection  of  the  stability  of  the 
questions  the  distributed  test  is  trying 
to  answer.  Indications  of  maturity 
are  identification  and  documentation 
of  measures  and  data  elements  that 
will  be  collected  from  the  test  and  the 
existence  of  a  federation  object 
model  (FOM),  an  interface  control 
document  (ICD),  or  a  federation 
execution  planners  workbook 
(FEPW)  for  the  distributed  test. 

Request  an  assessment  of  the 
maturity  of  the  requirements  from 
the  lead  engineer.  Adjust  risk  to 
all  elements  affected  by 
requirements  stability  according  to 
the  assessment. 

10 

Availability  of  player 
representations 

Medium/  Medium  -  This  element 
addresses  the  expected  availability  of 
each  player  during  integration  and 
execution.  Simulation/range  facilities 
that  support  more  than  one  customer 
during  the  critical  integration  and 
execution  periods  increase  the  risk  of 
schedule  conflicts  and  delays.  They 
also  risk  having  to  reconfigure  to 
meet  each  customer’s  needs. 
Reconfiguration  increases  the 
potential  for  inducing  problems 
during  integration  and  further  strain 
configuration  management. 

Ask  for  realistic  assessment  of 
each  player’s  availability  during 
the  critical  integration  and 
execution  periods.  Increase 
integration  and  execution  schedule 
risk  for  each  simulation/  range 
facility  supporting  multiple 
customers. 

11 

Schedule 

Medium/  Medium  -  This  is  a 
reflection  of  the  time  available  to 
complete  the  test  versus  the  initial 
estimate  of  the  time  needed  to 
complete  the  test. 

Review  this  list  for  factors  that 
increase  schedule  risk. 

12 

Experience  in  ADS/ 
similarity  to  previous 
distributed  tests 

Medium/  Low  -  This  is  a  correction 
factor  to  account  for  experience  and 
design  reuse. 

Increase  schedule  and  cost  risk  for 
no  experience  and  little  similarity 
to  previous  ADS  tests.  Do  not 
correct  for  extensive  ADS 
experience  or  complete  reuse  of 
previous  ADS  architecture  since 
estimates  should  be  more  realistic. 

13 

Implement  ADS 
network 

Low/Low-High  -  This  is  the  cost  of 
acquiring  and  operating  network 
links  and  standard  wide  area  network 
(WAN)/local  area  network  (LAN) 
equipment.  This  cost  is  not  difficult 
to  estimate,  but  recurring  costs  for 
long  term  tests  could  be  large. 

Commercial  providers  and 
communications  support  units  can 
provide  costs  given  the  number 
and  location  of  sites  to  be 
supported. 
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Impact 

Rank 

Cost  Factor 

Risk  to  Estimate/Expected 
Impact  to  Total  Cost 

Recommended  Estimator 
Response 

14 

Design  and  develop  test 
control  center 

Low/Low  -  This  is  the  cost  of 
acquiring  and  implementing  a  test 
control  center.  This  cost  is  not 
difficult  to  estimate  after  the  test 
control  concept  has  been  developed. 

The  lead  engineer  working  with 
communications  support  personnel 
should  be  able  to  develop  a  control 
concept  and  associated  parts  list. 

15 

Implement  network 
instrumentation 

Low/Low  -  This  is  the  cost  of 
acquiring  and  implementing 
instrumentation  to  measure  latency, 
bandwidth,  link  health,  and  other 

ADS  network  measures.  Player 
instrumentation  is  included  in  other 
elements. 

The  lead  engineer  working  with 
communications  support  personnel 
should  be  able  to  identify 
instrumentation  requirements  and 
develop  an  associated  parts  list. 

16 

Implement 
configuration 
management  system 

Low/Low  -  This  element 
acknowledges  the  increased  demands 
on  configuration  management  of 

ADS  over  traditional  testing.  Poor 
configuration  management  will 
increase  the  integration  schedule. 

Increase  integration  schedule  risk 
for  players  that  do  not  have 
demonstrated  strong  configuration 
management  practices. 
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Appendix  B  -  Guide  to  the  VV&A  of  an  ADS-Enhanced  Test  Environment 


The  verification,  validation  and  accreditation  (VV&A)  of  an  advanced  distributed  simulation 
(ADS)-enhanced  test  environment  should  proceed  hand  in  hand  with  the  design  and  development 
of  the  environment.  This  guide  to  the  W&A  of  an  ADS-enhanced  test  environment  will 
associate  the  appropriate  VV&A  activities  with  the  activities  contained  in  the  ADS-based  test 
planning  and  implementation  process.  More  details  on  the  VV&A  methodology  for  ADS-based 
tests  are  available  in  a  JADS  special  report.3 

The  High  Level  Architecture  (HLA)  Federation  Development  and  Execution  Process  (FEDEP) 
model  lists  six  steps  needed  to  develop  and  execute  an  ADS-based  test.  It  should  be  noted  here 
that  these  six  steps  might  involve  loop-backs  as  the  development  and  execution  of  the  test 
proceeds.  When  a  loop-back  occurs,  the  corresponding  verification  and  validation  (V&V)  will 
be  repeated  if  changes  are  made  to  the  design  or  execution  of  the  test.  The  six  steps,  as  listed  in 
Section  3.0  of  the  main  report,  are 

•  Step  1:  Define  Distributed  Test  Objectives 

•  Step  2:  Develop  Conceptual  Model 

•  Step  3:  Design  Distributed  Test 

•  Step  4:  Develop  Distributed  Test 

•  Step  5:  Integrate  and  Test  Architecture 

•  Step  6:  Execute  Distributed  Test  and  Analyze  Results 

An  ADS-based  test  has  the  requirement  that  the  ADS-enhanced  test  environment  and  its 
components  must  be  accredited  before  the  test  may  be  executed.  Since  the  accreditation 
authority  is  in  the  test  development  and  execution  chain,  a  step  needs  to  be  added  to  the  model 
between  Step  5  and  Step  6.  Step  5a  becomes  “accredit  ADS-enhanced  test  environment.” 

B.l  VV&A  Considerations  for  Step  1  (Define  Distributed  Test  Objectives) 

Early  in  this  step  the  test  manager  should  appoint  a  V&V  lead,  and  if  the  test  manager  is  the 
accreditation  authority,  an  accreditation  team  lead.  If  not  the  accreditation  authority,  ask  the 
accreditation  authority  to  appoint  an  accreditation  team  lead  to  be  a  member  of  the  integrated 
product  team  (IPT)  for  the  test. 

As  the  set  of  objectives  for  the  test  matures,  the  accreditation  team  lead  and  the  V&V  lead  should 
discuss  accreditation  requirements  and  how  the  V&V  will  be  conducted.  Broad  requirements 
and  acceptability  criteria  should  be  arrived  at  so  that  the  V&V  plan  can  be  developed.  Issues 
such  as  V&V  products  required  by  the  accreditation  team,  independent  V&V  versus  in-house 
V&V,  and  the  role  of  contractors,  if  any,  in  the  V&V  need  to  be  agreed  upon  and  documented.  It 


3  MAJ  Michael  Roane  and  Gary  Marchand.  JADS  Special  Report  on  Verification,  Validation  and  Accreditation  of 
Distributed  Tests,  available  from  the  Download  Area  of  the  JADS  web  site: 
http://www.jads.abq.com/html/jads/techpprs.htm. 
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is  also  helpful  for  them  to  identify  subject  matter  experts  (SMEs),  to  be  used  throughout  the  test 
and  V&V,  who  are  acceptable  to  both  the  test  manager  and  the  accreditation  authority. 

The  accreditation  team  lead  will  also  need  to  ask  the  accreditation  authority  if  an  accreditation 
team  from  within  the  accrediting  organization  (Joint  Advanced  Distributed  Simulation  [JADS] 
System  Integration  Test  [SIT]  and  Electronic  Warfare  [EW]  Test)  or  a  team  composed  of  SME 
from  outside  the  organization  (JADS  End-to-End  [ETEO  Test)  is  wanted.  Once  this  is 
accomplished,  write  an  accreditation  plan  describing  the  composition  of  the  accreditation  team, 
the  accreditation  objectives,  and  the  accreditation  requirements  or  acceptability  criteria. 

It  is  at  this  point  that  the  overall  V&V  plan  describing  the  tailored  process  that  will  be  used  by 
the  V&V  team  and  the  roles  of  the  members  of  the  IPT  should  be  written.  This  plan  will  be 
nonspecific  with  regard  to  actual  V&V  activities  as  the  actual  simulations,  nodes,  requirements, 
and  acceptability  criteria  for  the  ADS-enhanced  test  environment  have  yet  to  be  determined.  It 
will  match  major  V&V  events  with  major  test  events  and  describe  the  specific  V&V  events  that 
will  be  conducted  to  meet  the  previously  agreed  upon  broad  acceptability  criteria.  It  will  also 
identify  future  V&V  plans  to  be  developed  that  will  be  specific  in  nature.  This  plan  will  be  used 
to  cost  the  V&V  process. 

It  will  also  describe  in  some  detail  how  the  conceptual  model  will  be  validated,  as  this  validation 
will  occur  prior  to  firm  requirements  being  developed.  Finally,  it  must  identify  the  types  of  data 
required  to  evaluate  how  well  the  ADS-enhanced  test  environment  meets  the  test  objectives. 
This  must  be  done  at  this  step  so  that  the  test  designer  can  make  provisions  for  the  collection  of 
these  data  in  the  test  design.  This  plan  must  be  approved  by  the  test  manager,  after  review  by  the 
test  designer,  schedule  personnel,  resource  personnel,  and  the  accreditation  authority  or  a 
representative. 

Early  validation  of  the  conceptual  model  may  begin  during  this  step  as  SMEs  are  employed  to 
review  the  federation  objectives,  approach,  and  technologies.  The  efforts  of  these  SMEs  should 
be  documented  for  future  use.  Any  work  that  tries  to  determine  how  well  the  ADS-enhanced  test 
environment  will  represent  the  real  environment  should  be  considered  as  conceptual  model 
validation. 

B.2  VY&A  Considerations  for  Step  2  (Develop  Conceptual  Model) 

This  step  is  composed  of  three  activities  that  are  very  important  to  the  VV&A  of  an  ADS- 
enhanced  test  environment. 

B2.1  Activity  2.1  -  Develop  Scenario 

The  basis  for  the  V&V  of  the  scenario(s)  to  be  used  in  the  test  must  be  established  here.  The 
V&V  team  must  work  closely  with  the  scenario  developers  documenting  data  sources,  doctrinal 
sources,  and  fidelity  requirements.  The  requirements  for  the  V&V  of  the  scenario  should  have 
been  identified  in  the  V&V  plan  developed  during  the  previous  step,  to  include  the  accrediting 
authority  for  the  scenario. 
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B2.2  Activity  2.2  -  Perform  Conceptual  Analysis 

It  is  at  this  point  that  the  distinction  between  test  team  members  and  V&V  team  members 
becomes  fuzzy.  The  purpose  of  this  activity,  for  the  test  team  members,  is  to  ensure  that  the 
conceptual  model  maps  to  the  objectives,  scenario,  doctrine  and  tactics,  and  fidelity  requirements 
of  the  test.  In  other  words,  they  want  to  determine  how  well  the  federation  represents  the  real- 
world  test  environment  and  is  the  representation  adequate.  They  do  this  in  order  to  reduce  risk 
prior  to  proceeding  with  the  development  of  the  federation.  The  V&V  team  wants  to  do  the  same 
thing.  The  two  teams,  if  they  are  different  people,  should  work  together  within  the  IPT  structure 
to  ensure  that  this  activity  provides  the  required  degree  of  risk  reduction  to  the  test  manager,  as 
this  is  its  primary  purpose. 

With  respect  to  the  accreditation  of  the  ADS-enhanced  test  environment,  the  validation  of  the 
conceptual  model  is  a  freebie.  The  results  will  be  overshadowed  by  the  validation  of  the 
constructed  federation  and  the  accreditation  decision  will  be  based  upon  the  performance  of  the 
federation  not  a  conceptual  model.  As  stated  above,  conceptual  validation  is  conducted  in  order 
to  reduce  risk. 

B2.3  Activity  2.3  -  Develop  Distributed  Test  Requirements 

This  is  probably  the  most  important  activity,  with  respect  to  V&V,  in  the  test  planning 
methodology.  Requirements,  to  the  person  conducting  V&V,  are  like  specifications  to  a  person 
building  a  system.  They  dictate  what  the  federation  and  its  federates  have  to  do  to  satisfy  the  test 
objectives  and  measures.  As  stated  in  the  test  planning  methodology,  they  fall  into  several  broad 
categories.  These  categories  are  fidelity  requirements,  interaction  requirements,  latency 
requirements,  data  reliability  requirements,  and  data  analysis  requirements.  All  require  V&V 
effort  to  determine  how  well  the  federation  and  its  federates  meet  the  requirements. 

It  is  critical  that  the  V&V  team  be  a  part  of  the  IPT  and  participate  in  the  development  of  the 
requirements.  They  will  be  able  to  offer  valuable  insights  as  to  what  is  measurable  quantitatively 
and  what  must  be  measured  qualitatively.  It  is  equally  important  to  have  the  accreditation  team 
lead  present  because  it  will  be  the  responsibility  of  the  team  lead,  working  with  the  test  director 
and  the  accreditation  authority,  to  develop  the  acceptability  criteria. 

If  the  requirements  state  what  the  federation  and  its  federates  should  do,  the  acceptability  criteria 
state  the  minimum  acceptable  performance  of  the  federation  and  its  federates.  Ideally,  the  tester 
would  like  for  any  simulation  of  a  system,  or  process,  to  be  perfect.  Practically,  close  is  often 
good  enough.  The  acceptability  criteria  define  close.  If  the  federation  and  its  federates  fail  to 
meet  an  acceptability  criteria,  the  accreditation  team  has  the  option  of  recommending  that  the 
federation  be  modified  to  meet  the  criteria  or  that  the  test  proceed  at  the  observed  level  of 
performance. 

Once  the  requirements  and  acceptability  criteria  are  identified,  the  detailed  V&V  plan(s)  may  be 
developed  stating  the  specific  V&V  activities  to  be  performed  during  the  design,  development, 
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integration  and  testing  of  the  federation.  These  plans  must  be  approved  by  the  test  manager,  after 
review  by  the  test  designer,  schedule  personnel,  resource  personnel,  and  the  accreditation  team. 

B3  VV&A  Considerations  for  Step  3  (Design  Distributed  Test) 

The  V&V  team  will  need  to  work  closely  with  the  designer  during  this  phase  as  it  functions  as  a 
quality  control  team  for  the  test  manager.  Its  responsibility  is  to  verify  that  HLA  federation  and 
federate  rules  have  been  followed,  where  applicable,  and  that  the  federation  and  federates 
conforms  to  the  object  model  template.  The  team  should  also  document  any  deviations  from  the 
HLA  rules  and  the  reason  why  the  deviations  were  required. 

The  V&V  team  will  also  verify  that  the  detailed  design  will  meet  the  acceptability  criteria  arrived 
at  during  the  last  step  and  that  the  objectives  of  the  test  will  be  met  by  the  proposed  design.  All 
or  most  of  these  tasks  will  also  be  conducted  by  the  design  team.  The  V&V  team  provides  a 
second  set  of  eyes,  with  a  slightly  different  perspective,  to  ensure  that  the  design  is  adequate  and 
to  reduce  risk  within  the  test.  The  use  of  modeling  tools,  to  model  the  federation  and  its 
federates,  is  recommended  as  a  V&V  or  design  tool  during  this  step. 

Finally,  the  V&V  team  will  verify  that  adequate  provisions  are  made  in  the  design  for  the 
collection  of  data  to  V&V  the  final  federation  as  assembled. 

B.4  VV&A  Considerations  for  Step  4  (Develop  Distributed  Test) 

This  step  is  a  continuation  of  the  previous  step  and  the  V&V  effort  will  be  directed  toward 
reducing  risk.  As  federates  are  modified,  they  will  be  tested  to  ensure  they  meet  requirements 
and  that  they  continue  to  be  HLA  compliant. 

The  major  emphasis  at  this  point  should  be  on  ensuring  that  the  federation  will  satisfy  the  test 
requirements  and  objectives  and  less  on  standards  compliance.  The  purpose  of  V&V  is  not  to  act 
as  a  standards  policeman.  Given  the  choice  of  a  successful  test  meeting  all  requirements  and 
objectives  or  blind  adherence  to  a  standard,  most  test  managers  will  decide  on  the  former. 
Deviations  from  standards  should  be  documented  and  reported  to  the  standards  authority  along 
with  the  “fixes”  employed  to  make  the  test  environment  work. 

As  they  are  identified,  components  of  the  federation,  to  include  hardware  items,  may  be  procured 
and  tested  for  HLA  compliance  and  functionality.  Items  such  as  routers,  crypto  devices,  and 
leased  communication  lines  may  also  be  tested  and  characterized  within  the  federation 
configuration.  It  is  recommended  that  a  test  bed  federation  with  dummy  federates  be  established 
using  all  the  actual  hardware  components  before  moving  to  the  next  step.  The  dummy  federates 
receive  and  publish  data  in  the  same  manner  and  at  the  same  rate  as  the  actual  federates  are 
expected  to  perform.  This  allows  the  test  bed  to  experience  the  same  interactions  and  data  flow 
as  the  actual  federation.  Interactions  and  data  flow  may  be  determined  from  the  model  of  the 
federation  recommended  in  the  previous  step. 
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Using  the  test  bed,  hardware  upper  limits  can  be  established  for  data  flow  within  the  federation, 
and  the  effects  of  changes  to  the  architecture  and  data  flow  can  be  easily  observed.  Proper 
documentation  is  essential  at  this  point,  especially  if  changes  are  made  to  the  architecture  that 
deviates  from  the  HLA  standard. 

B.5  W&A  Considerations  for  Step  5  (Integrate  and  Test  Architecture) 

Up  to  this  point,  the  purpose  of  the  V&V  conducted  was  to  reduce  risk.  The  V&V  conducted 
during  this  step  will  be  used  to  convince  the  accreditation  team  and  the  accrediting  authority  that 
the  ADS-enhanced  test  environment,  or  federation  if  HLA  is  used,  meets  the  acceptance  criteria 
or  at  least  performs  well  enough  to  proceed  with  the  test. 

V&V  activities  must  be  coordinated  with  the  integration  and  functionality  activities  conducted  by 
the  test  team.  Ideally,  no  additional  resource  requirements  because  of  V&V  will  be  placed  on 
this  step.  The  purpose  of  this  step  is  to  assemble  the  ADS-enhanced  test  environment  and 
determine  how  well  it  functions.  Functionality  is  measured  in  terms  of  how  well  does  the 
environment  meet  its  requirements.  V&V  also  need  to  determine  and  document  how  well  the 
environment  functions.  Whether  it  is  called  functionality  testing  or  V&V  is  immaterial,  both 
have  the  same  objective. 

Compliance  issues  should  have  already  been  answered  and  documented.  That  is  why  the  use  of 
the  previously  mentioned  test  bed  is  recommended.  Deviations  from  HLA  standards,  in  order  to 
make  the  ADS-enhanced  test  environment  meet  requirements,  will  be  more  readily  accepted  than 
an  HLA-compliant  test  environment  that  does  not  meet  requirements. 

Occasionally,  a  system  or  system  of  systems  will  be  tested  that  is  so  complex  that  the  ADS- 
enhanced  test  environment  only  exists  during  actual  test  events.  An  example  is  the  JADS  ETE 
Test  conducted  by  JADS  that  contained  an  aircraft  flying  at  35,000  feet  over  Texas.  Because 
these  events  are  often  costly  in  terms  of  time  and  resources,  the  test  manager  is  reluctant  to 
allocate  one  of  these  events  to  V&V.  Sufficient  integration  and  functionality  testing  can  take 
place  on  the  ground  to  satisfy  the  test  manager  that  the  ADS-enhanced  test  environment  has  a 
high  chance  of  success  during  an  actual  test  trial.  V&V  can  also  be  conducted  of  the  almost  real 
ADS-enhanced  test  environment  to  answer  how  well  the  environment  meets  the  test 
requirements.  However,  if  the  test  environment  exists  only  during  the  test,  the  V&V  conducted 
are  again  risk  reduction  events. 

B.5.1  Step  5a  -  Accredit  ADS-Enhanced  Test  Environment 

This  is  probably  the  easiest  step  in  the  process,  provided  the  previous  VV&A  activities  have  been 
performed  as  suggested.  You  normally  do  not  arrive  at  this  point  in  the  process  unless  the  ADS- 
enhanced  test  environment  is  performing  well  enough  to  satisfy  the  acceptability  criteria 
established  by  the  accreditation  authority.  In  addition,  if  you  have  fully  involved  the 
accreditation  team  in  the  planning  and  conducting  of  the  V&V  events,  they  already  know  how 
well  the  ADS-enhanced  test  environment  meets  the  acceptability  criteria.  This  is  especially  true 
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if  you  provide  them  with  the  supporting  documentation  as  it  is  produced,  rather  than  dumping  all 
of  it  on  them  just  prior  to  the  accreditation  meeting. 

Obviously,  if  the  actual  test  environment  has  not  been  verified  and  validated,  as  described  in  the 
last  step,  the  accreditation  of  the  actual  test  environment  cannot  take  place  until  after  the  test 
event(s)  have  been  conducted.  The  accreditation  will  be  based  on  the  actual  test  performance  of 
the  environment  and  will  confirm  that  the  data  collected  during  the  test  are  valid.  The 
accreditation  team  should  meet  prior  to  the  test  and  consider  the  V&V  conducted  during  the 
integration  and  testing  of  the  “near”  actual  test  environment.  They  can  still  recommend 
proceeding  to  test  with  the  stipulation  that  the  final  accreditation  will  not  occur  until  after  the 
test. 

If  this  is  not  acceptable,  the  accreditation  authority  will  have  to  be  willing  to  accept  the  allocation 
of  the  first  test  event  to  V&V  with  the  understanding  that  if  the  test  environment  is  subsequently 
accredited,  the  data  collected  during  the  V&V  are  valid  SUT  data. 

B.6  VV&A  Considerations  for  Step  6  (Execute  Distributed  Test  and  Analyze 
Results) 

Why  V&V  after  the  ADS-enhanced  test  environment  has  already  been  accredited?  Well,  simply 
put,  most  ADS-enhanced  test  environments  are  so  complicated  that  the  only  way  you  can  tell 
they  are  functioning  correctly  is  to  repeat  some  of  the  V&V  activities  during  the  test  event.  In 
order  to  prove  that  you  have  collected  valid  test  data,  you  must  establish  that  the  ADS-enhanced 
test  environment  was  meeting  its  requirements. 
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Appendix  C  -  Test  Control  Structures  for  Distributed  Testing 


C.l  Test  Control  Mechanism 

A  test  control  mechanism  is  a  critical  element  of  every  test  event.  Control  is  exercised  by  a  test 
director  who  is  the  person  responsible  for  test  execution.  In  live  testing,  control  is  often 
exercised  from  a  range  control  facility.  In  developmental  test  and  evaluation  (DT&E),  control 
may  be  exercised  from  a  laboratory  control  center.  Regardless  of  the  type  of  testing,  a  workable 
test  control  capability  is  a  fundamental  requirement  for  each  and  every  test  event.  Distributed 
test  events  are  no  exception  to  the  rule,  they  also  require  a  test  control  mechanism,  but  the 
distributed  environment  demands  some  differences  in  approach. 

C.2  Local  Control  Mechanisms 

In  a  distributed  test  architecture,  participating  facilities  are  scattered.  The  facilities  may  be 
represented  in  any  of  the  three  forms  of  advanced  distributed  simulation  (ADS):  live,  virtual,  or 
constructive.  The  facilities  may  include  hardware-in-the-loop  (HWIL),  man-in-the-loop  (MITL), 
digital  system  models  (DSMs),  simulators,  or  simulations,  or  combinations  of  such  capabilities 
and  assets. 

At  each  node  in  the  distributed  architecture  are  people  who  are  expert  in  the  maintenance  and 
operation  of  the  capabilities  of  that  node.  There  are  also  hardware  and  software  capabilities  to 
observe  and  report  on  the  operations  of  local  devices.  At  each  node  there  needs  to  be  a  local 
control  mechanism  with  a  single  person  responsible  for  test  support  operations.  The  control 
mechanism  incorporates  a  level  of  local  decision  authority  appropriate  to  the  expertise  at  the 
node.  Only  the  local  experts  know  the  resident  systems  characteristics  and  behaviors  well 
enough  to  make  on-the-spot  judgments  about  status,  performance,  and  risk.  When  live  players 
are  involved  at  a  node,  then  it  is  clear  that  safety-related  decision  making  must  take  place  at  the 
node. 

Local  control  mechanisms  should  be  designed  perform  two  functions.  The  first  is  to  provide  the 
ability  to  start,  stop,  or  hold  local  activities  in  response  to  situations  where  safety,  damage 
prevention,  security,  or  other  kinds  of  critical  elements  are  involved.  The  second  is  to  provide 
continuous  and  timely  status  reporting  to  a  central  agency  responsible  for  overall  test  control. 
Some  of  the  hardware  and  software  necessary  to  support  reporting  will  not  be  a  part  of  the  local 
system.  The  test  program  supported  will  have  to  provide,  install,  and  sometimes  operate,  the 
equipment  and  programs  required  to  interface  with  the  distributed  status  and  control  system.  The 
equipment  can  range  from  new  communications  lines  to  routers,  switches,  data  loggers,  and 
network  interface  units. 
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C.3  Central  Control  Mechanism 


As  with  traditional  test  control,  distributed  test  control  is  exercised  by  a  test  director.  Like  a 
traditional  counterpart,  the  distributed  test  director  is  responsible  for  test  execution.  Unlike  a 
traditional  counterpart  however,  the  distributed  test  director  has  less  hands-on  control  of  the 
environment  because  the  elements  of  the  environment  are  geographically  dispersed. 

It  is  both  sensible  and  necessary  that  any  node  retain  the  authority  to  start  or  stop  local  operations 
in  response  to  circumstances  at  its  site.  It  is  the  test  director,  however,  who  decides  whether  to 
stop  or  modify  ongoing  test  activity.  As  an  example,  it  is  possible  that  a  breakdown  at  a  single 
test  node  may  not  be  fatal  to  continued  testing  —  it  may  force  a  change  in  test  event  type  but  not 
force  a  halt  to  testing.  It  is  the  central  authority,  the  test  director,  who  has  to  make  test  start,  test 
stop,  and  test  modification  decisions  relevant  to  the  test  event. 

The  timely  and  continuous  status  reporting  from  the  local  nodes  is  a  critical  underpinning  of  the 
central  control  mechanism.  Much  of  the  reporting  can  be  automated.  Commercial-off-the-shelf 
(COTS)  software  is  available  to  monitor  the  condition  of  each  piece  of  hardware  in  the  system, 
data  flow  in  each  leg  of  the  architecture,  and  the  real-time  values  of  error  sources  such  as  latency. 

The  central  control  facility  must  have  the  display  and  communications  capabilities  to  know  total 
system  health  in  real  time.  Total  system  health  includes  not  only  the  status  of  the  real,  virtual, 
and  constructive  players,  but  the  data  processing  and  collection  system,  and  the  system 
synchronization  mechanism. 

C4  Blending  Local  and  Central  Control  Mechanisms 

The  challenge  in  constructing  a  test  control  mechanism  for  a  distributed  test  lies  in  the  proper 
mix  of  central  and  local  control.  Many  test  and  evaluation  (T&E)  organizations  are  used  to 
having  absolute  control  over  their  own  resources,  and  they  tend  to  bridle  at  the  notion  that  control 
may  reside  outside  their  bailiwick.  The  ranges  are  particularly  sensitive  to  what  they  see  as 
intrusions  over  their  control  of  live  assets. 

If  only  one  range  with  live  players  is  incorporated  into  an  architecture,  then  it  is  technically 
feasible  (although  it  may  not  be  desirable  for  other  reasons)  to  locate  the  test  control  center 
within  the  range  control  facility.  If  more  than  one  range  is  involved,  then  there  have  to  be  some 
accommodations.  The  guiding  rule  for  blending  areas  of  control  is  “don’t  do  anything  stupid.” 

The  ranges  have  the  mechanisms  to  control  operations  at  their  locations,  so  they  should  control 
operations.  That  includes  clearing  range  activity,  controlling  actions  on  the  range,  making  safety 
and  weather  decisions,  aborting  missions,  and  the  like.  Having  said  that,  a  range  is  not  in  a 
position  to  know  the  real-time  status  of  every  item  in  a  distributed  architecture.  Only  the  test 
control  center,  wherever  it  is  located,  has  what  might  be  termed  a  global  picture.  The  test 
control  center  controls  the  test  and  that  includes  starting  the  test,  terminating  the  test,  or  directing 
modifications  to  ongoing  test  activities.  The  direction  of  modifications  —  such  things  as 
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changing  routes  or  timing,  altering  engagements,  etc.,  is  an  area  where  there  is  potentially  a  direct 
interface  with  the  ranges.  The  implementation  of  the  test  director’s  modifications  may  very  well 
have  to  be  implemented  at  the  sites  where  there  are  live  players.  To  put  it  another  way,  direction 
to  modify  “operations”  may  have  to  be  passed  to  the  local  controlling  agency  (range  control 
center,  for  example)  for  execution  rather  than  directly  to  the  players. 

An  area  of  concern  within  the  control  function  is  configuration  management.  There  should  be  a 
clear  understanding  among  all  parties  that  configuration  control  is  a  function  of  the  test  director 
and  that  local  changes  in  configuration  require  the  test  director’s  approval. 

C.5  Designing  a  Test  Control  Mechanism: 

The  preceding  paragraphs  in  this  section  were  intended  to  describe  the  conceptual  view  of  test 
control  within  a  distributed  architecture.  From  a  process  point  of  view,  the  construction  of  a  test 
control  mechanism  for  a  distributed  test  is  requirements  driven,  and  the  requirements  are  test 
specific. 

The  test  planning  and  implementation  methodology  described  in  Section  3  of  this  document 
provides  the  blueprint  for  a  given  test  or  test  event.  Once  the  test  is  defined  to  a  level  where  the 
objectives,  the  measures,  and  the  supporting  data  elements  are  known,  then  Step  1  of  test  control 
design  can  be  initiated. 

C.5.1  Step  1.  Laying  Out  the  Instrumentation 

When  all  the  data  elements  are  identified  in  the  test  plan,  it  is  possible  to  lay  out  the 
instrumentation  and  data  collection  plans.  The  instrumentation  and  data  collection  structure  are 
important  to  the  control  design  because  the  test  control  center  must  be  capable  of  monitoring  the 
status  of  those  elements  in  detail.  The  test  director  must  ensure  that  the  entire  instrumentation 
and  data  collection  system  are  functional  before  directing  the  start  of  testing.  If  a  piece  of 
instrumentation  or  data  collection  equipment  goes  down,  it  may  be  necessary  to  stop  a  test  in 
progress.  In  many  cases,  the  instrumentation  and  data  collection  system  will  be  imbedded  in 
existing  local  area  networks  (LANs)  at  the  various  nodes.  In  some  cases  it  may  be  necessary  to 
create  new  LANs. 

C.5.2  Step  2.  Laying  Out  the  Wide  Area  Network  (WAN) 

In  Step  1,  the  information  about  the  data  elements,  their  points  of  origin,  and  the  technical 
characteristics  of  the  data  streams  are  developed.  That  information  defines  the  basic  connectivity 
required  in  terms  of  number  of  nodes,  data  types,  and  bandwidth.  In  effect,  the  information 
defines  the  outline  of  the  WAN.  (See  Sections  3.3.1  and  3.3.2  for  specific  methodology  on 
network  design  and  development.)  The  bandwidth  requirements  identified  in  this  phase  are 
tentative.  There  will  be  additional  bandwidth  required  to  support  redundancy  (if  needed)  and 
control  functions. 
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C.5.3  Step  3.  Laying  Out  the  Test  Control  Center 

When  there  is  a  good  understanding  of  the  network  layout,  it  is  possible  to  initiate  design  of  a  test 
control  center.  The  displays  must  accommodate  representation  of  instrumentation  and  data  flow 
throughout  the  distributed  architecture.  Network  monitoring  and  recording  capabilities  must  be 
identified.  Voice  and  video  communications  links  must  be  established  to  meet  control  needs. 
Logging,  recording,  storage,  and  playback  devices  must  be  identified.  Network  synchronization 
capability  needs  to  be  addressed.  Quick-look  analysis  capabilities  need  to  be  defined  and 
supporting  equipment  identified.  When  the  technical  capabilities  and  the  supporting  equipment 
are  well  defined,  then  manning  and  space  needs  can  be  addressed.  Step  3  defines  the 
requirements  for  the  test  control  center. 

C.5.4  Step  4.  Choosing  a  Location  for  the  Test  Control  Center 

In  theory,  a  test  control  center  could  reside  anywhere  in  a  linked  architecture.  In  practice,  cost 
will  weigh  heavily  in  the  decision  about  location.  If  the  test  director  owns  or  has  access  to  a 
range  control  facility,  such  a  site  may  be  an  ideal  location  for  a  test  control  center.  Range  control 
facilities  have  most  of  the  control  functions  required  for  distributed  testing  already.  They  have 
displays,  processing  power,  voice  (and  in  some  cases  video)  communications,  and  some  data 
recording  and  storage  capability.  When  the  requirements  identified  in  Step  3  are  overlaid  on  the 
capabilities  of  an  existing  control  facility,  then  the  necessary  modifications  can  be  identified  and 
costed,  and  that  data  can  be  weighed  in  the  site  selection  process. 

C.5.5  Step  5.  Integrating  Local  Control  Nodes  with  the  Test  Control  Center 

Once  a  test  control  site  is  selected,  integration  of  the  entire  distributed  architecture  can  begin. 
Facilities  participating  in  the  distributed  test  are  providing  access  to  resources  they  operate  on  a 
routine  basis.  Within  their  own  facilities,  or  sometimes  their  LANs,  they  have  their  own  control 
mechanisms.  The  modes  of  communications  among  the  test  control  center  and  the  control 
centers  at  the  nodes  have  to  be  addressed  on  a  case-by-case  basis.  Is  voice  a  requirement?  Do  I 
need  an  open  line?  How  about  video?  When  such  questions  are  answered,  then  the  impact  of  the 
solutions  needs  to  be  fed  back  into  the  WAN  and  LAN  design  requirements. 

Aside  from  the  technical  issues  of  integration,  there  are  bureaucratic  issues  to  contend  with.  It 
may  be  desirable  to  establish  memoranda  of  understanding  or  agreement  between  the  test  director 
and  the  supporting  facilities  to  lay  out  the  specifics  of  control  sharing.  In  the  execution  of  this 
process  step,  it  is  also  advisable  to  address  the  need  for  assigning  test  force  personnel  to  each  of 
the  nodes.  If  there  are  such  needs,  then  the  agreements  need  to  address  them. 

C.5.6  Step  6.  Documenting  Control  Processes  and  Procedures 

Once  the  control  mechanism  architecture  is  defined,  it  is  possible  to  move  on  to  the  final  step  in 
the  process.  Complex  tests  are  like  complex  weapons  systems  in  at  least  one  regard.  Checklists 
and  procedures  are  required.  Rules  of  engagement  have  to  be  documented.  Start-up  procedures 
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need  to  be  developed  and  implemented.  Players  in  the  control  process  need  to  be  specifically 
identified  and  their  responsibilities  defined  and  documented.  Emergency  procedures  have  to  be 
defined.  The  list  goes  on.  Each  test  will  have  different  sets  of  procedures,  but  there  is  a  common 
thread  in  all  tests.  Procedures  must  be  developed,  documented,  and  rehearsed.  As  changes 
occur,  and  they  will,  the  changes  must  be  captured  in  the  documentation. 

C.6  Summary 

This  part  of  the  document  doesn’t  contain  great  detail.  Individual  tests  have  individual  control 
mechanism  needs.  The  common  characteristic  of  all  distributed  tests  is  a  dispersed  player  set. 
That  dispersion  forces  the  test  planner  to  develop  a  control  mechanism  which  provides  a  sensible 
blend  of  local  and  central  control. 
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